Add paging support for URL/HTML

[TobiasNx]

We can only open single URLs but cannot page trough multiple URLs.
This is connected to #460 since it is relevant for GET and POST with APIs.

[dr0i]

To be discussed @fsteeg and @blackwinter :
we could enhance HttpOpener, but I think better would be a module on its own, like:

$urlToRetrieveAllTotalItemsAsJson
| open-http
|decode-json(recordpath="pagination.totalItems")
| page-http(url="$baseUrlWithPaginationParmeter", size="100")
|print
;

Identified with @TobiasNx at least 3 different types of pagination:

1. with "total items" and a "size" (see example above)
2. using $urlWithPageParameter , where the value of "page" is incremented as long as no 404 appears
3. using a (next/resumption) token

(1) and (2) are already demanded by oersi.
The httpPager would be capable of doing all 3 kinds of paging types. Using (1) would be independent of the serialisation of the return of the API (json/xml/text). (3) could maybe also be used as a naive OAIPMH harvester.

[fsteeg]

I don't quite understand how your example(s) would work. I think I'd need a concrete API / URL / input data (for the different kinds of pagination).

Perhaps starting with the OERSI example that I used in my first / WIP approach to pagination support: https://gitlab.com/oersi/oersi-etl/-/commit/60ba8d70b1dd728006cd07485b632ecef961e98d

Did you see the Swissbib approach: https://github.com/linked-swissbib/swissbib-metafacture-commands#open-multi-http

And I had this general thought of using URL globbing for paging somehow: https://everything.curl.dev/cmdline/globbing

[dr0i]

re swissbib: is just a very sad way to do it, because you have to know how many data there is to begin with and set these values via parameters. My approach would parse the actual totalItems and pass them to the pager.

Your approach is stuck to Json as result. Also it seems to make use of sitemaps to get all resources - this would be another type "4" (as I understand we want to switch from sitemap to using an API)

re globbing: this is type (2).

Concrete example for using an API (type (1) ) (enabled by #463):

'{"criterias": [], "facettes": []}'
| open-http(url="https://www.zoerr.de/edu-sharing/rest/search/v1/queriesV2/-home-/-default-/ngsearch?maxItems=10&skipCount=0&propertyFilter=-all-", method="post")
| decode-json(recordpath="pagination.totalItems") // => 1348
| page-http(url="$baseUrlWithPaginationParmeter", size="100")
|print
;

[TobiasNx]

one thing I think is difficult is, that the input within MF for page-http would vary for every paging type. This would make this modul complex in it self.

[blackwinter]

I don't think I have much to add at this point. Except maybe that type 3 could also be useful for Elasticsearch pagination (search_after and/or scroll).