API client pagination - can't use 'x-total-pages' for GitLab.com

[cdboer]

The api client does not request more than the first page for some resources from projects hosted at GitLab.com.
The pagination approach relies on the key x-total or x-total-pages being present in the request response headers.

The GitLab API doc's state the following concerning x-total and x-total-pages:

For performance reasons, if a query returns more than 10,000 records, GitLab doesn’t return the following headers:

x-total.
x-total-pages.
rel="last" link.

If both keys are missing from response headers, gitlab2prov naivly assumes that there is only one page of the requested resource.
Not all GitLab.com projects are affected. Updates will follow.

See also this section in the official GitLab documentation.

[cdboer]

This issue requires a rework of the gitlab2prov.api.request_handler class. We can no longer rely on the x-total pages key and can therefore not craft all request coroutines 'a priori'. A request coroutine could create new ones if needed and add them to a centralized queue if there are more pages left.

class RequestHandler:

    def __init__(self):
        self.client = None
        self.queue = []

    async def request_page(self, url, page_number):
        url = url.update_query({"page": page_number})
        async with await self.client.get(url) as resp:
            json = await resp.json()
            next_page = resp.headers["x-next-page"]
        if page_number < next_page:
            self.queue.append(self.request_page(url, next_page))
        return json

[cdboer]

In the future we should move to KeySet-based pagination, as GitLab is slowly converting their API endpoints to support it. This should speed up response times for requests for bigger projects.