Investigate pagination and e-tag headers for SCM providers

[cmanallen]

[linear]

ENG-6903

[cmanallen]

@jacquev6 I compelled Claude to implement this change so ignore the volume of code (didn't cost me a thing). d59b44b (types.py)

I've added a pagination scheme for passing opaque cursors from the response to the next request. I'm at the end of my workday so I'm uninterested in thinking about this anymore today lol. Let me know what you think (comment or stand up).

I've also added cache headers in a RequestOptions dict. That should be mostly uncontroversial.

[jacquev6]

(I'm writing this before reading @cmanallen's comment to avoid confirmation bias)

Technical context

GitHub

Generally, pagination on GitHub is based on links returned in the Link header. It contains links for the next page (and previous, first and last ones).

Most, if not all, collection paths (e.g. issues) support offset-based pagination with page and per_page parameters for arbitrary page access.

GitLab

All but the "users" endpoint support offset-based pagination with page and per_page parameters as well.

Pagination Link header is returned as well.

Use cases

(A) Similar interface with and without RPC

Client code should look the same wether they are using the SCM platform through the RPC or not.

(B) Always return raw data

The raw data returned by each request made to the SCM provider should be accessible to the client code.

(C) List all items

When a single function in the client, called once, needs all items at once. Most common case?

sequenceDiagram
    participant C as Client
    participant SCM as SCM platform
    participant P as Provider
    C->>SCM: Get all items
    SCM->>P: Get first page
    P->>SCM:
    loop Until last page
        SCM->>P: Get next page
        P->>SCM:
    end
    SCM->>C:

Loading

(D) List first few items

When the client code only needs some items. E.g. when filtering repositories by name, it might be okay to ask the user to type a few more letters to find the repo they are looking for if it does not appear at first.

sequenceDiagram
    participant C as Client
    participant SCM as SCM platform
    participant P as Provider
    C->>SCM: Get some items
    SCM->>P: Get the first page
    P->>SCM:
    SCM->>C:

Loading

(E) Iterate items until arbitrary condition

When the client is looking for a specific item but their selection criteria aren't implemented by the provider's API.

sequenceDiagram
    participant C as Client
    participant SCM as SCM platform
    participant P as Provider
    C->>SCM: Get first items
    SCM->>P: Get first page
    P->>SCM:
    loop While item not found
        C->>SCM: Get next items
        SCM->>P: Get next page
        P->>SCM:
        SCM->>C:
    end

Loading

(F) Expose items, paginated

When the client wants to display all items in the user's browser, displaying each as soon as it's available for maximum reactiveness.

In that case, because the client is a stateless web server, pagination information must be transmitted to the browser.

sequenceDiagram
    participant B as Browser
    participant C as Client
    participant SCM as SCM platform
    participant P as Provider
    C->>SCM: Get first items
    SCM->>P: Get first page
    P->>SCM:
    C->>B:
    loop Until last page
        B->>C: Get next page
        C->>SCM: Get next page
        SCM->>P: Get next page
        P->>SCM:
        SCM->>C:
        C->>B:
    end

Loading

Possible interfaces

get_items() -> Iterator[Item]

Pagination is completely hidden from the client.

(pro) They don't have to think about it. They can't forget it.

(pro) So (C) is easy.

(pro) (E) as well.

(con) Some calls to next take longer to complete than other (when the SCM platform has to fetch the next page), and this is hidden from the client. Leaky abstraction.

(con) For (B), access to page-level raw data is complicated at best. We could attach the page-level raw data to each item of the page. Or we could accept to split it into item-level raw data, forfeiting access to hypothetical data besides the list of items on the page.

(con) (D) can be implemented using itertools.islice, but if the client really wants just one page, they have to know the page size. Very leaky abstraction.

(con) (F) is nearly impossible. We could imagine letting the client marshall the state of the iterator, send it to the browser as an opaque token, get it back, but... yuk. Plus the token would be large because the state of the iterator actually contains at least the remaining items on the current page. Double yuk.

get_items_pages() -> Iterator[Page]

Pagination is explicit.

(pro) This makes it hard to forget.

(pro) Each Page can have its raw data for (B).

(neutral) For (C) and (E), client code needs a double loop, similar to using itertools.groupby. We could provide an adapter to reduce this to a single loop on items when they don't need page-level data.

(pro) (D) is easy.

(neutral) For (F), we could add next_page_token to the Page object and accept it as a first_page_token to get_item_pages. Client would still have to call next explicitly to get only the first page, so we might decide to additionally provide get_items_page(page_token) instead.

get_items_page() -> Page

(con) This has the same characteristics as get_items_pages() -> Iterator[Page] but without the easy-to-use iterator interface.

[cmanallen]

@jacquev6 Context. You mentioned GitHub and GitLab using offset pagination, which is true for 99% of our present cases, but GitHub's GraphQL APIs use cursor based pagination: https://docs.github.com/en/graphql/guides/using-pagination-in-the-graphql-api#requesting-a-cursor-in-your-query. So we'll need to support cursor-based pagination as well.

[cmanallen]

@jacquev6 I'm broadly against implementing an Iterator approach in the SCM (though that doesn't mean I can't be convinced). Especially Iterator[Item]. I think iterators should be handled by a utility function iter_scm_action(callable, args) with the iter_scm_action handling pagination parameters transparently for callers.

The SCM actions should expose the ability to finely control pagination in case there's some product need to do so.