Impagination
Put the fun back in lazy, asynchronous, paged, datasets.
Impagination is a lazy data layer for your paged records. All you provide Impagination is the logic to fetch a single page, plus how many records you want it to pre-fetch ahead of you.
Impagination is built using an event-driven immutable style, so it is ideal for use with UI frameworks like Ember, Angular, or React. That said, it has zero dependencies apart from JavaScript, so it can be used from node as well.
Installation
npm install impagination
Upgrading
If you are upgrading Impagination
to the 1.0
release, consider checking out the Migration Guide
Usage
To get started, create a dataset. There are only two required parameters
fetch
, and pageSize
:
; let dataset = pageSize: 5 // num records per page loadHorizon: 10 // window of records to keep (default: pageSize) { // How to `fetch` a page statstotalPages = 4; // Returns a `thenable` which resolves with page's `records` return $; } {} // invoked whenever a page is unloaded {} // filters `records` whenever a page resolves { // invoked whenever a new `state` is generated datasetstate = nextState; };
Calling new Dataset()
will emit a state
immediately. However this state
will be empty.
datasetstatelength //=> 0; let record = datasetstate; // Empty RecordrecordisRequested //=> falserecordisPending //=> falserecordisResolved //=> falserecordcontent //=> null
To start fetching pages and build the state
, we need to start reading from an offset. To do this, we will update the dataset's readOffset
.
dataset;
With a pageSize
of 5, this will immediately call fetch twice (for records 0-4, and 5-9),
and emit a new state indicating that these records are in flight.
dataset;datasetstatelength //=> 10; // Records 0-9 are Pending Recordslet record = datasetstate;recordisRequested //=> truerecordisPending //=> truerecordisResolved //=> falserecordcontent //=> null
Load Horizon
How did it know which records to fetch? The answer is in the
loadHorizon
parameter that we passed into the constructor.
We set the loadHorizon
to 10. This tells the dataset,
that it should keep all records within 10 of the
current read offset loaded. That's why it fetched the first two
pages.
I hope this ASCII Dataset adds some clarity:
dataset = ...; // builds the dataset below
Read
Offset
┃
┃
<──────────Load Horizon──────────>
┃
▼
┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┐
* * * * * * * * * *
◇ ─ ─ ─ ─ ◇─ ─ ─ ─ ┘
│ │ │
p0 p1 length = 10
ASCII Legend:
`xx` - unrequested record
`*` - pending record
`n` - record inex
`◇` - page boundary
Resolving Asynchrnous Pages
Once the asynchronous fetch
for a page resolves, the
dataset will emit a new state
with the updated resolved records.
Continuing our previous example, we assume the the request on page 0
resolves and the
request on page 1
is not yet resolved. That state will still contain the resolved records as well the pending records.
datasetstatelength //=> 20; // Assumes the page `0` resolves and page `1` is pendinglet record = state;recordpageoffset = 0;recordisPending //=> falserecordisResolved //=> truerecordcontent //=> { name: 'Record 3' } record = staterecordpageoffset = 1;recordisPending //=> truerecordisResolved //=> falserecordcontent //=> null
Another interesting thing that happened here is that the length of the dataset has also changed.
datasetstatelength //=> 20 (stats.totalPages: 4, pageSize: 5)
That's because the stats
parameter that is passed into our example
fetch function tells our dataset there are 5
total pages in our dataset.
This value allows the fetch function to optionally
specify the total extent of the dataset if that information is
available. This can be useful when rendering native scrollbars or
other UI elements that indicate the overall length of a list. If
stats
are never updated, then the dataset will just expand
indefinitely. Now our state looks like this:
Read
Offset
┃
┃
<──────────Load Horizon──────────>
┃
▼
┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ── ─ ┐
0 1 2 3 4 * * * * * xx xx xx xx xx xx xx xx xx xx│
◇ ─ ─ ─ ─ ◇ ─ ─ ─ ─ ◇ ─ ─ ─ ─ ─ ─ ◇ ─ ─ ─ ─ ─ ─ ┘
│ │ │ │ │
p0 p1 p2 p3 length = 20
We have records 0-4, whilst records 5-9 are in flight, and records 10-20 have yet to be requested.
//from the last ◇ page (p3)record = state;recordisRequested //=> falserecordisPending //=> falserecordcontent //> null
Dataset API
There are a number of public impagination
functions which we provide as actions to update the dataset.
Updating the Dataset
Actions | Parameters | Description |
---|---|---|
refilter | [filterCallback] | Reapplies the filter for all resolved pages. If filterCallback is provided, applies and sets the new filter. |
reset | [offset] | Unfetches all pages and clears the state . If offset is provided, fetches records starting at offset . |
setReadOffset | [offset] | Sets the readOffset and fetches records resuming at offset |
Updating the State
Actions | Parameters | Defaults | Description |
---|---|---|---|
post | data, index | index = 0 | Inserts data into state at index . |
put | data, index | index = state.readOffset | Merges data into record at index . |
delete | index | index= state.readOffset | Deletes data from state at index . |
setReadOffset Example
Let's say the we change our viewport to item 2 in our UI. We want to tell impagination to move the read head to offset 2 with a call to dataset.setReadOffset(2)
. This will immediately emit a new state
that looks like this:
Read
Offset
┃
┃
<──────────Load Horizon──────────>
┃
▼
┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ── ─ ┐
0 1 2 3 4 * * * * * * * * * * xx xx xx xx xx│
◇ ─ ─ ─ ─ ◇ ─ ─ ─ ─ ◇ ─ ─ ─ ─ ─ ◇ ─ ─ ─ ─ ─ ─ ┘
│ │ │ │ │
p0 p1 p2 p3 length = 20
You'll notice that the page at offset p2 has now been requested because
it contains records that fall within the load horizon. The page at
p1
is still pending, but now p2
is as well. What happens if the
request for p2
resolves before the request for p1
? In that case,
the dataset emits this state:
Read
Offset
┃
┃
<──────────Load Horizon──────────>
┃
▼
┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ── ─ ── ─ ┐
0 1 2 3 4 * * * * * 10 11 12 13 14 xx xx xx xx xx│
◇ ─ ─ ─ ─ ◇ ─ ─ ─ ─ ◇ ─ ─ ─ ─ ─ ─ ◇ ─ ─ ─ ─ ─ ─ ┘
│ │ │ │ │
p0 p1 p2 p3 length = 20
In this way, impagination is resilient to the order of network requests because the records are "always available" and in their proper order, albeit in their unrequested, pending, or resolved states.
//records on p2 are now availablerecord = state;recordisResolved //=> truerecordcontent //=> 10 //records on p1 are still pendingrecord = state;recordisResolved //=> falserecordisPending //=> truerecordcontent //=> null
Filtering Records
We fetch records using an immutable style, but we often require filtering by mutable values in our dataset. To enable filtering, pass a filter callback
to impagination
as you would to Array.prototype.filter()
. The filters are applied as soon as a page is resolved. To filter a page at other times in your application see refilter
.
Here we filter by records whose content contains an even number
let dataset = ... // filter() function which returns only _even_ records { return content % 2 === 0 };
Read
Offset
┃
┃
<──────────Load Horizon──────────>
┃
▼
┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ── ─ ── ─ ┐
0 2 4 * * * * * ** ** ** ** ** xx xx xx xx xx│
◇ ─ ─ ─ ─ ◇ ─ ─ ─ ─ ◇ ─ ─ ─ ─ ─ ─ ◇ ─ ─ ─ ─ ─ ─ ┘
│ │ │ │ │
p0 p1 p2 p3 length = 18
// Finding even numbered recordsrecord = state;recordisResolved //=> truerecordpageoffset //=> 0recordcontent //=> 2statelength //=> 18 (stats.totalPages: 4, pageSize: 5, rejected records by filter: 2) //records on p1 are still pendingrecord = state; // The record at index 3 now exists on p1recordisResolved //=> falserecordisPending //=> truerecordpageoffset //=> 1recordcontent //=> null
Impagination and Immutability
In the mutable style of reactivity, you listen to events that report
what changed about a datastructure, and then you're left to realize
the implications of that change in your internal data structures (such
as changing a record from isPending
to isResolved
). By contrast,
Impagination uses an immutable style.
In Impagination, each event is the fully formed datastructure in its entirety. This eliminates all guesswork and ambiguity from what the implications are so that you, the developer, have to do less work to maintain consistency.
What this means in practice is that each of the states observed by
the observe
function are unique structures that are considered
immutable. Each one stands alone and will continue to function
properly even if you discard references to all other states and the
dataset object itself. Furthermore, altering them will have no effect
on neither prior nor subsequent states.
You may be asking, is it not wasteful to recreate an entire potentially
infinite data structure with every state transition? The answer is
that each state is lazy and stores as little information as it needs
to provide its API. The state
contains lazy array interfaces for pages
and records
.