Fix api pagination - retrieve all resource pages from GitLab projects/project repositories #43
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Rename GitlabAPIClient to GitlabClient Export URL building into its own method in GitlabClient Rework URL building, splitting path and query building into their own methods Rename alru_cache to just cache as it does not represent an asynchronous least recently used cache and is used as a "one time store" cache Add the ability to change the project path of the GitlabClient without having to reinstantiate the URLBuilder Rely on the group_by function to sort results added to gitlab2prov/utils/__init__.py
The group_by function is used to match page results back to their appropriate request url's. It is also used to regroup request results such as issue note awards. For example: The GitlabClient is supposed to return a list of issue note awards for each issue of a project, but it can only request a list of awards for each issue note. To regroup the awards such that there is a list of awards for each issue and not for each issue note, one can use the group_by function and pass a group_size_index that reflects how many issue notes there are on each inbdividual issue.
Reason: no longer in use
Rewrite of RequestHandler: Requests are now queued on demand. Added pagination fallback to the key 'x-next-page' to not solely rely on 'x-total-pages'. Add initial exception system, see TODO notes. Exceptions are supposed to be handled in the GitlabClient methods.
Makes it more obvious that the request queue does not hold request urls but rather request coroutines that have to be awaited to be executed. The queued coroutines are awaited in a asyncio.gather statement.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #41.
Mentions #39 #42.
Changes proposed in this pull request:
./ideadirectory to.gitignore