Offloading Data Baggage with Livewire

In Hoarding Order with Livewire we implemented a client paginated table that relies on data accumulation. Instead of waiting for an entire dataset to load, the table periodically received and accumulated smaller portions of the dataset using Livewire:poll.

Data accumulation is the process of storing more data on top of an initial set of data retrieved. Our table’s accumulated data will eventually grow to complete the entire dataset, and so will the space in memory it takes up.

Today, let’s dive headfirst into optimizing data accumulation with a focus on “Offloading data-accumulated baggage.”

You can check out the full repository here.

What is data-accumulated baggage?

The table we’ve set up in Hoarding Order with Livewire used client side pagination to easily paginate grouped rows of data. Instead of downloading an entire dataset, it fetched and accumulated parts of the entire dataset in the background using Livewire:poll.

This approach made pagination easy and lowered the bottleneck on processing large datasets. However, it also has a glaring drawback.

As the polling mechanism accumulates more data for the table component, the larger the memory sent back by the server and received by the client browser, and consequently:

1. The longer the time it takes for the client to download data from the server
2. The larger the memory space taken up by our users’ devices

We are left with “data baggage” we must not allow to get out of hand!

Offloading the Baggage

Data accumulation can leave unnecessary, memory-heavy baggage. Today, we’ll clean up both server and client baggage’s we’ve accumulated so far:

1. Offload data accumulation from server and assign the role to the client
2. Reset accumulated data in client when it becomes irrelevant to the user

Offloading Data Accumulation to the Client

Previously, to accumulate data using Livewire, we periodically added data to a public Livewire array $dataRows. We used a Livewire:poll element that periodically called nextPageData:

// resources\views\livewire\article-table.blade.php
<div wire:poll.5s>
  {{ $this->nextPageData() }}
  {{ $this->dispatchBrowserEvent('data-updated', ['newData' => $dataRows]); }}
</div>

nextPageData eventually calls addListToData, a method that adds more data to $dataRows:

// app\http\Livewire\ArticleTable.php
public function addListToData($noneSubList)
{
    $subList = $this->getSubRows($noneSubList);
    foreach( $noneSubList as $item ){
+        $this->dataRows[] = $item;
        ...<redacted>...  
    }
}

As our server’s $dataRows grows with accumulation, so does the time it takes for devices to download it.

Client devices can download 10 rows easily in milliseconds. However, it can take more than a second for devices to download rows counting to 1000 and above. Yikes!

Luckily, this accumulation in the server is not necessary at all.

Our client already holds data accumulated by the server in a separate, client-side attribute myData. The client updates myData by listening to the data-updated event which sends back $dataRows as newData.

// \resources\views\livewire\article-table.blade.php
window.addEventListener('data-updated', event => {
   myData = event.detail.newData;
   refreshPage();
});

We also don’t want to be redundant by having the accumulated data in both server and client. So, let’s keep accumulation strictly client side.

1) From the server, clear $dataRows—the attribute that accumulates data—for every new batch of data we’re processing in app/http/Livewire/ArticleTable.php:

public function addListToData($noneSubList)
{
+   // Let's clear the data list we have
+   $this->dataRows = array();
    $subList = $this->getSubRows($noneSubList);
    foreach( $noneSubList as $item ){
            $this->dataRows[] = $item;
            ...<redacted>...  
    }
}

Our server can breathe easier with smaller datasets to store and send out. And now, our client can more quickly download those datasets.

All that’s left now is to move accumulation logic to our client:

2) Revise /resources/views/livewire/article-table.blade.php to merge its accumulated data myData with new lists received from the server:

window.addEventListener('data-updated', event => {
-   myData = event.detail.newData;

+   // Append new dataset to myData list
+   myData.push(...event.detail.newData)

   // Reload the table display
   refreshPage();
});

That’s it for offloading server data baggage! When working with Livewire, keep your server responses lightweight. This means no data accumulation from the server when the client can do it instead.

Offloading Client Data-Baggage

Data accumulation was removed from the server above, but our client still holds accumulated data.

To alleviate this burden left on our client, let’s keep only necessary data stored in memory. We can do this by clearing the client’s accumulated list every time our user requests to view a subset of our data.

As an example, if we have a table of bookmarks—urls saved from different sources—when the user asks to view bookmarks from “www.pacocha.com”, the only subset of data the client needs to store are those bookmarks containing that substring. The client wouldn’t need other irrelevant data in the table, and so, we can safely clear the data it has accumulated so far, and start off new with relevant results.

This offloading allows our users’ devices to breath easier from each reset. As an overview, we’ll set up five things to offload our client data:

1. A public Livewire attribute $filters which will receive parameters from our client using Livewire:model.
2. A search bar mapped to $filters[search] that will let us know specific keywords our user wants to filter our data with.
3. A Livewire updated hook that detects changes on $filters , triggering a call to our Livewire component’s initializeData method that sends back filtered data and a reset flag to our client through the data-updated event.
4. A revision on our client listener to data-updated that will clear its accumulated myData when it receives a reset=true flag.
5. Finally, a static method filterQuery in our model which will receive $filters and apply client parameters to our database queries.

View

Let’s equip our client’s view, resources/views/livewire/article-table.blade.php, with a search bar for sending search queries, and a logic for clearing its accumulated data.

1) From our view, we can bind a search bar with a public Livewire attribute using wire:model:

#  resources/views/livewire/article-table.blade.php
+ <div> 
+   <input type="text" wire:model.debounce.500ms="filters.search" placeholder="Search" class="bg-gray-50 border border-gray-300">
+ </div>

2) Next, let’s revise our client listener on the data-updated event to clear its accumulated list myData whenever it receives a reset=true flag from the server:

#  resources/views/livewire/article-table.blade.php
<script>
...

window.addEventListener('data-updated', event => {
+ // Clear list when a reset is received and return to page 1
+  if( event.detail.reset ){
+    myData = [];
+    page   = 1;
+  }

   // Append new dataset to myData list
   myData.push(...event.detail.newData)

   // Reload the table display
   refreshPage();
});

That’s it for our view. Conditionally clearing myData is the core of offloading data baggage from the client!

What’s left below is to filter data based on user request. Below, we’ll update our controller to respond to changes on the public attribute mapped to our search bar, $filters.

Controller

After revising our View above, we should now have a search bar to send $filters[search] requests to our server.

In our Controller, /app/http/Livewire/ArticleTable.php, let’s detect changes on $filters, apply those filters to our queries, and send back a filtered batch of data along with a reset flag to the client.

Detecting Filters

From our controller, we can add our public attribute $filters and listen to any changes on it through Livewire's updated hook.

1) Add a public $filters attribute in /app/http/Livewire/ArticleTable.php.

+ public $filters;

// Override this to initialize our table 
public function mount()
{
+   // Let's use this for search, filter, and sort logic  
+   $this->filters = [];
    ...
}

2) Detect changes on $filters through Livewire's updated hook. Any user input on $filters should trigger a call to our Livewire component’s initializeData method.

// Re-initialize our data for all changes made on $this->filters
+ public function updatedFilters()
+ {
+    $this->initializeData();
+ }

The initializeData method gets a new set of data for us. It eventually passes its new dataset to addListToData, which is in charge of sending a properly revised, ordered list of data to our client.

3) Pass a reset=true flag between the initializeData to addListToData methods

public function initializeData()
{
    $noneSubList = $this->getBaseQuery()
    ->limit($this->pagination*2)
    ->get();

-   $this->addListToData( $noneSubList );
+   $this->addListToData( $noneSubList, true );
}

4) And finally, from our addListToData method pass the ordered dataset and the reset flag to the client through a dispatchBrowserEvent on the data-updated event

- public function addListToData($noneSubList)
+ public function addListToData($noneSubList, $resetClientList=false)
{
    ...<redacted logic here>...

-   $this->dispatchBrowserEvent('data-updated', ['newData' => $this->dataRows]); 
+   $this->dispatchBrowserEvent('data-updated', ['newData' => $this->dataRows, 'reset'=>$resetClientList]);  
}

Completing Controller revisions 1 to 4 above enabled detection of user-changes to $filters attribute. The changes above would clear the client’s accumulated myData list with each user search, but it still does not process the filtering logic.

Applying Filters

Applying filters to our data is very easy!

We’ll later create a static function in our model to receive and process requests from $filters. For now, let’s make our final changes to our controller to chain that static filter method in our existing model queries:

1) We have two methods in our /app/http/livewire/ArticleTable.php that’s in charge of querying our Article model. So let’s chain our filter query there:

// Base Query for none-Sub data retrieval
public function getBaseQuery()
{
     // Quickly refresh the totalRows every time we check with the db
-    $this->totalRows = Article::query()->count();
+    $this->totalRows = Article::filterQuery($this->filters)->count();

     // Return none-Sub rows to avoid duplicates in our $dataRows list
-    return Article::query()->whereNull('lead_article_id');
+    return Article::filterQuery($this->filters)->whereNull('lead_article_id');
}

...

// Get's Sub data for a list of possible Lead rows
private function getSubRows($noneSubList)
{
    $idList = [];
    foreach($noneSubList as $item){
        $idList[] = $item->id;
    }

-   return Article::query()
+   return Article::filterQuery($this->filters)
    ->whereIn('lead_article_id', $idList)
    ->get();
}

Model

Finally, let’s add in a static method in our model to apply the $filters we receive.

Revise /App/Models/Article.php and add a method where we can apply filters with.

# /App/Models/Article.php

+ public static function filterQuery(array $filters)
+ {
+   $query = Article::query();

+   // Search
+   if( isset($filters['search']) ){
+       $query = $query->where(function($query) use($filters){
+           $searchString = '%'.$filters['search'].'%';
+           $query->where('url', 'like', $searchString );
+           $query->orWhere('source', 'like', $searchString );
+       });
+   }   

+   return $query;
+}

That’s it!

In one sitting we were able to create one method in our model to apply $filters, use that filter method throughout our model queries, detect and respond to user input on $filters through Livewire's updated hook, bind user search on $filters through Livewire:model, and finally breathe easy with offloading (irrelevant) client data baggage!

Retrospect

In the first part of this two-post series on paginating tables with grouped rows, we saw that we can do client-pagination without downloading entire datasets. With Livewire, we can now easily poll for more data to accumulate instead of waiting for an entire dataset to load.

This day, amidst the heavy baggage we’ve steadily accumulated, we saw that it’s okay to offload data baggage. We can still have client-pagination on accumulated data.

We’ll just remove accumulation from the server side to avoid sending large datasets for the client to download. And, with a little mix of reloading from the server when the time calls, our client can breathe easy with clearing its baggage as well.

It’s okay to remove baggage—in the right place, time, and manner.