[RFC] support pagination using cursors

[yonahforst]

I'm in the middle of writing a client for AWS DynamoDB. One of the problems is that the DB uses cursor-based pagination (is that what it's called?)

I'm using a custom reducer to keep track of the previous + next keys, but I need a way to pass those from the <Pagination/> component to the client.

At first I thought it would be a good idea to allow the page param to be an object, but that got messy quickly. there are several places where it's assumed to be an integer.

However, filters can be objects and maybe that's a better fit anyway. What do you guys thinking about exposing setFilters to the <Pagination/> component?

It's working great for me and I have a PR lined up to submit.

[fzaninotto]

Good idea. And while we are at it, we could expose a setPerPage method, allowing the pagination to change the number of rows per page.

[yonahforst]

Well, perhaps I spoke too soon when I said 'working great'.

I don't think my current implementation is suitable as generic DynamoDB client that anyone can use (which I would very much like it to be).

My main problem is that I don't know the prev/next cursor keys until the server responds with the contents of the current page. Somehow I need to pass those key to <Pagination/>. Now I'm doing it with a custom reducer and making it a ConnectedComponent but it's really ugly and requires a bunch of config.

I'm actually considering moving back to page and perPage and just maintaining a list of ids locally to the client. (DynamoDB cursors are just ids, so it's easy to figure out prev/next)

What would you guys recommend?

another option would be just to have a custom saga that updates the filters with the next/prev keys upon CRUD_GET_LIST_SUCCESS and then read them in <Pagination/> without connecting it

[yonahforst]

using page and perPage is working well for me.
https://github.com/abiglobalhealth/aor-dynamodb-client

still working on supporting CREATE, UPDATE, DELETE

[mrcasablr]

@yonahforst repo link is a 404

[yonahforst]

Whoops. Had to make it public. Thanks for catching that

[djhi]

Can you make a PR to add it to the ecosystem section in the documentation ?

[yonahforst]

Done. Pr #1518

[garethsb]

We'd love to see cursor-based pagination support in React Admin.

As @yonahforst says above, what's needed is for a way for the dataProvider to return the prev/next (and optionally first/last) cursors/URLs with the returned data for GET_LIST and GET_MANY_REFERENCE, e.g. { data: [...], cursors: { prev: ..., next: ..., } } rather than { data, [...], total: 42 }.

How this is best hooked up to the Pagination component is debatable. Would it would be best for the Pagination component to to have access to the whole data (I don't think it does now?) and do e.g. setPage(data.cursors.next)?

Just noticed #1787, which proposes something similar. @fzaninotto, am I on the right track?

[fzaninotto]

Pagination does not receive the data key of the dataProvider response, you're right. But it receives the resource data from the store (an array of objects already fetched) and the list of ids that is deduced from the dataProvider data, so it's feasible to recompose the initial data variable.

Changing the GET_LIST return value format is a breaking change that we won't address in the near future, so I recommend that you write a custom Pagination component using the cursor/URL form the data.

[garethsb]

Thank you very much for the response.

But it receives the resource data from the store (an array of objects already fetched) and the list of ids that is deduced from the dataProvider data, so it's feasible to recompose the initial data variable.

I think you're suggesting adding the cursors to one or more of the individual returned resources' data? In many APIs with cursor-based pagination, the response does not return a cursor per resource, but only cursors that identify the positions at the start/end of the current page. There may be cursors for the previous or next page, even when the page itself is empty. So e.g. associating the cursors with one of the resources may be possible sometimes but not in the general case.

I recommend that you write a custom Pagination component using the cursor/URL form the data.

Do you suggest this should use the dataProvider directly, rather than try to (ab)use the setPage mechanism?

Thanks!

[isumix]

The offset & limit or page & perPage based pagination is broken by design, unless we never change our data. So I'd suggest streamline this for react-admin.

Meanwhile, I think adding filter like {after: 'some_id'} could help.

@fzaninotto, what do you think about it?

PS: there is a comprehensive pagination spec I like.

[jtomaszewski]

Problem with filters is that they are memoized if the user goes out and back to the list view. I managed to put the page and cursor param in the filters, and it works; but now if the user goes back to the list view, they end up on the "last page they've been browsing", instead of starting with page 1 again.