fix: serialize pipeline matrix legs to prevent secondary GitHub rate limits - #154
Draft
cursor[bot] wants to merge 1 commit into
Draft
fix: serialize pipeline matrix legs to prevent secondary GitHub rate limits#154cursor[bot] wants to merge 1 commit into
cursor[bot] wants to merge 1 commit into
Conversation
…token Add max-parallel: 1 to the run_pipelines matrix so github, slack, citations, and newsletter never hit api.github.com concurrently on the same GH_TOKEN_STATS_PAGE token. Concurrent github + citations legs tripped a secondary rate limit (issue #153) even when the primary quota showed 4970/5000 remaining. Document the serialization constraint in pipeline/AGENTS.md for future GitHub-API-calling pipelines. Co-authored-by: Edmund Miller <edmundmiller@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #153
Problem
The
githubandcitationsmatrix legs inrun_pipelines.ymlboth authenticate toapi.github.comwith the sameGH_TOKEN_STATS_PAGEtoken. When they run concurrently, citations' ~200-request burst (2nextflow.configlookups per nf-core pipeline) collides with github's org-repo crawl on the same token, tripping GitHub's secondary rate limit — even when the primary quota still shows thousands of requests remaining.Run 32287740582 is the canonical example:
/rate_limitpreflight read4970/5000remaining, then the very next real request (GET /orgs/nf-core/repos) 403'd with a reset time that didn't match the hourly bucket.Fix
max-parallel: 1to therun_pipelinesmatrix strategy sogithub,slack,citations, andnewsletternever race each other against the same GitHub token.pipeline/AGENTS.mdso future GitHub-API-calling pipelines stay in the serialized matrix.Trade-off
Pipeline legs now run sequentially instead of in parallel, so total CI wall time increases. This is intentional — the alternative is intermittent secondary rate-limit failures that abort the github crawl mid-run.