Skip to content

Deal with potential pagination problems #42

@carlosms

Description

@carlosms

There can be problems importing paginated resources if some of them are deleted while paginating.
Storing all of them on memory could be also expensive.

#32 (comment)

Imagine you have 101 repos.
GET the first page, and start processing one by one their issues and PRs.
Meanwhile a repo gets deleted, and the total number of repos is 100.
After a while we finish processing all the repos in the first page, and we GET page 2. But now github will say that there is no page 2, and we missed the processing of 1 repo.

#32 (comment)

[...] we still get error anyway if something got deleted during import.

#32 (comment)

[...] this may impact on memory consumption, [...] it would be better to avoid storing the repos in an array and just call s.doRepo for each repo for each page.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions