API: Apply limit by default rather than returning all records

[matthew-white]

By default, sending a request for submissions will return all submissions even if there are very many of them. This issue is to apply a limit by default. In other words, we should paginate by default and apply a page size (of 100 records?).

Not all endpoints support pagination, and this issue does not aim to change that. An example of that is the form list endpoint. This issue is to apply a default page size for the following endpoints that already support pagination:

• Submission list (OData)
• Submission list (REST API)
• Entity list (OData)
  • I don't think the REST API currently supports pagination.

[matthew-white]

Submission list (REST API)

As we work on this part, we should also take a look at this issue: #1099. We should make sure we've documented that pagination is available in the REST API.

[matthew-white]

Notes from dev meeting today

@lognaturel says: We should think through the implications of this change for existing integrations, since it would be a breaking change.

@sadiqkhoja says: The OData metadata document can specify configuration for the default pagination. PowerBI would look at that configuration, but custom integrations might not.

[brontolosone]

Paginating through an ordered collection can only get you the complete collection if the collection is immutable (which it isn't, in our case) or if the cursor used is sound (a changed_at time-based cursor isn't, because of a. transaction visibility and b. non-uniqueness).

In other words, currently if you page through, say, the submission collection, you may see duplicates, and also, you may miss submissions. So it's impossible to reliably synchronize (which you may want to do for any analysis pipeline).

In other words, currently an unpaged index dump (/v1/projects/{projectId}/forms/{xmlFormId}/submissions) is the only way to get a consistent snapshot of the collection state (which can then serve as a basis to retrieve submission definitions & their attachments).
So if we turn on pagination by default, we need to make sure it's still possible to get such a consistent view!

One way to do that is to use the new eventstamp as a cursor. But that presupposes that on the receiving side, the entries are interpreted as mutations rather than as unique submissions — in other words, you might see a submission ID pop up twice up while paging through, which means the submission was updated in some respect during the paging. And then the receiving side needs to propagate that change into its representation of the collection. Basic synchronization stuff, but conceptually different from what the current paging does.