Repository navigation
fix(ci): base-churn fan-out saturates the runner queue, so PRs are neither red nor green #15869
Description
Measured while diagnosing why seven open PRs are simultaneously not red and not green.
The board's state is queue saturation, not failure
Of the 100 most recent workflow runs:
| State | Count |
|---|---|
| queued | 61 |
| completed / success | 32 |
| in_progress | 4 |
| completed / skipped | 3 |
| completed / failure | 0 |
| completed / cancelled | 0 |
61% of recent runs have not started. Nothing is failing. Every PR that looks stuck is waiting for a
runner, and every summary view renders "waiting" and "not passing" identically.
Where the volume comes from
Each push fans out to 20-29 workflow runs:
29 runs 039439a4
25 runs 004b639d
24 runs 702439f4
20 runs b7cb37f3
And a base commit does not produce one push. The auto-update bot merges base into every open PR
branch, so one commit to Dev_new_gui becomes one new head per open PR, each of which is a fresh
fan-out:
- 14 commits to
Dev_new_guiin the last 12 hours — one every ~51 minutes - 7 open PRs
- ~25 runs per head
That is on the order of 14 × 7 × 25 ≈ 2,450 runs in 12 hours from base churn alone, before anyone
pushes any actual work. The arrival rate is set by base churn multiplied by the number of open PRs,
while the service rate is fixed by the runner pool — so the queue grows with the product, and every
PR that stays open makes it worse for every other PR.
Why it is self-sustaining
A PR needs an uninterrupted window long enough to complete ~25 runs. Base moves every ~51 minutes, and
each move issues a new head for that PR, which supersedes the in-flight work
(cancel-in-progress: ${{ github.event_name == 'pull_request' }}, e.g. migration-gate.yml:47). So a
PR only lands if it completes inside a gap in base churn — and merging is the only thing that
reduces the fan-out, because it is the only thing that closes a PR.
Nothing green means nothing merges; nothing merging means the fan-out never shrinks. The two
things anyone can do to help — pushing a fix, updating a branch — both add runs.
What this is not
- Not flaky tests: zero failures in the sample.
- Not concurrency cancellation as the dominant effect: zero cancelled in the sample. Individual
cancellations do happen on rapid re-push (three on one PR earlier today), but they are not what is
holding the board. - Not review capacity: several PRs have zero unresolved threads and are still not mergeable.
Acceptance criteria
- The runs a base commit fans out to are counted and recorded, so the multiplier is a known number
rather than an inference from run listings - The auto-update bot does not re-issue a head for every open PR on every base commit — batch,
rate-limit, or update only PRs that are actually being merged. This is the multiplier, and
reducing per-head cost does not address it - Runs whose result cannot change are not scheduled: a base merge that touches no path a workflow
cares about should not re-run it on every open PR - A queued-run depth metric is visible somewhere a session can read, so "the PR is stuck" can be
distinguished from "the queue is deep" without listing 100 runs - Measured after the change: median time from head-created to all-required-contexts-reported, on a
day with comparable base churn
The way this gets closed without being fixed
By speeding up individual workflows. Every run getting 20% faster does not change an arrival rate that
exceeds the service rate — it moves the saturation point slightly and leaves the mechanism intact. The
second criterion is the one that matters: the multiplier is the defect, and it is set by how many
heads get created, not by how long each run takes.
Activity
mrveiss commentedon Sep 6, 2026
The queue is now delaying a production repair, which changes what this issue is
Since filing, the depth has grown rather than drained:
Still zero failures. The board has one PR with a failing check; everything else is waiting.
The evidence that makes this a correctness problem rather than a throughput one: code-sync on the
live host is broken by
66b46ebb7d(becomeon aTaskInclude, which Ansible rejects at parse time,aborting the whole self-update play). The repair is PR #15870 — 9 insertions, 6 deletions, one
file — and it is
BLOCKEDbehind the same queue as everything else, with 27 checks unreported, 3cancelled and 5 succeeded.
So the mechanism described above is not costing minutes on feature work. It is holding a one-line
production fix behind roughly 25 workflow runs that exist because the fan-out re-issues heads for every
open PR on every base commit — and the repair cannot merge until they report, because nothing merges
before green.
That is worth stating plainly in the acceptance criteria, so the fix is not scoped as an optimisation:
base-churn fan-out — a priority lane, a reduced required set for a defined class of fix, or a
queue that drains oldest-first rather than being continually re-fed. Which mechanism is a
design decision; the criterion is that "the host is broken and the fix is queued behind 2,000
runs of branch maintenance" has an answer.
Why this is not solved by merging faster
Merging is the only action that reduces the fan-out, and merging requires green, and green requires
the queue to drain. The one lever anybody has is gated behind the thing it would fix. That is the
argument for treating the multiplier — heads created per base commit — as the defect, rather than any
property of the individual runs.
mrveiss commentedon Sep 6, 2026
Saturation hides wrong verdicts, which is a correctness cost the earlier arguments do not cover
The two arguments above are both about delay — the repair waits, the multiplier grows. This one is
not, and it is the reason this cannot be closed as an optimisation.
A PR that is about to go red is indistinguishable from one that is still queued. Both show
unreported required contexts against a board where most runs have not started. So a wrong "this is
ready, merge it when the contexts report" does not surface as a wrong call — it surfaces as normal,
for as long as the queue stays deep.
That happened here, today, on the production repair this queue is already holding:
A reviewer (me) checked whether removing
become: truefrom aninclude_taskswould trip a repoguard, searched two directories, found nothing, and reported "no guard objects — merge it the
moment its contexts report."
The guard was in a third directory, beside the code it guards
(
autobot-slm-backend/tests/test_service_units_self_sufficient_14693.py:59), and demandsbecomeon every top-level task in that file. Measured against both trees:
Had that go-ahead been acted on, the PR would have gone red — and read as "still queued",
because that is what everything reads as right now.
The error was mine and it was caught by a peer re-running the assertion. The point for this issue is
what would have happened if it had not been: the queue converts a wrong green into a silent wait.
Normally a bad go-ahead is corrected within minutes by the check that fails. Under saturation there is
no such feedback, because "not passing yet" is the expected state of every PR on the board.
Why this does not depend on the queue being this deep
The delay arguments need the queue to exceed some tolerance. This one only needs it to be deeper
than a reviewer's patience — long enough that "unreported" stops prompting anyone to look. That
threshold is much lower than 73 queued runs, and it is reached long before anyone would describe CI as
broken.
It also means faster runners do not fix it. Halving every run halves the delay and changes nothing
about the indistinguishability, because the two states still render identically for however long the
window lasts.
Criterion this adds
"running", and "will not run", from the PR view, without listing workflow runs by hand.
"Waiting" and "not passing" rendering the same is what lets a wrong verdict sit unchallenged —
and it is the state every PR on this board is in right now.
mrveiss commentedon Sep 6, 2026
Correction: the arrival model in this issue is wrong
The issue says one commit to
Dev_new_guibecomes one new head per open PR, and derivesbase-churn × open-PRs × ~25 runs. That multiplier does not exist..github/workflows/auto-update-pr-branches.yml:136setsMAX_BRANCH_UPDATES: '1', and line 249breaks the loop on it:
So the bot refreshes exactly one behind PR per base push, skipping any that are mid-run. The
≈ 2,450 runs in 12 hoursfigure and the "arrival rate is base-churn × open-PRs" framing are bothwrong and should be read as withdrawn.
What the measurements still support, since they were counted rather than derived:
and 20-29 workflow runs per push (29, 25, 24, 20 across four consecutive heads). The dominant term
is therefore workflows-per-push, plus one bot refresh per base commit — not a per-PR fan-out.
What this changes:
every base commit", describes behaviour that is already capped. It should be rewritten against the
real term: ~25 runs for a push that may touch one file, and how many of those can be skipped when
nothing they gate has changed.
which still costs ~25 runs whenever it is pushed or refreshed; it does not cut a per-commit
multiplier, because there isn't one.
What this does not change: the queue counts, zero failures, the production repair sitting behind
the queue, and the camouflage argument — that a PR about to go red is indistinguishable from one still
queued. None of those depend on the arrival model. The camouflage criterion in particular is about how
state is rendered, not about how work arrives.
The mechanism was asserted to me and I built the issue on it without running the one-line grep that
would have grounded it — the same failure the camouflage comment above documents me making in a code
review two hours earlier. Recording it here rather than quietly editing the body, because an issue
whose evidence changed should show that it changed.
mrveiss commentedon Sep 6, 2026
A live instance of the camouflage, from a different direction: duplicate check-runs per name
Measured on PR #15864's current head. Seven check names report more than once:
And the two entries for one of them disagree about whether it has run:
So "did
code-qualitypass?" has two answers on one head, and which one a reader gets depends ona resolution rule nobody stated. A gate that takes the first match, or treats
SKIPPEDas a pass,calls this PR ready on a check that never ran — while the run that will actually decide it is still
in progress.
This is the same defect as the earlier argument in this issue and it does not need the queue at all.
The earlier form was "waiting" and "not passing" render identically. This form is one check name
carrying contradictory verdicts simultaneously, which is worse in one respect: the reader is not
looking at an absence and guessing, they are looking at a stated
SKIPPEDand taking it at facevalue.
Both come from the same root — the fan-out creates multiple runs per name per head, from path-filtered
workflows that emit a required context whether or not their real work applies (#11346, #13862 made
that deliberate, and correctly so). The cost lands on whoever reads the result.
Criterion this adds
applied consistently: the latest conclusive run per name decides,
SKIPPED/NEUTRAL/CANCELLEDare not verdicts, and a name with an in-flight run is pending, neverresolved by an earlier
SKIPPED. Any gate, dashboard, or session reading check state mustagree, or the same head will be called ready by one reader and blocked by another.
Worth noting the correct rule is already known and applied by at least one reader — the loop's gate
reports
pend=for exactly this reason. What is missing is that it is implicit: nothing stops thenext tool, or the PR page itself, resolving the same data differently.
mrveiss commentedon Sep 6, 2026
Refinement to the duplicate-verdict criterion above
I wrote it as "the resolution rule is stated and applied consistently". That is not enough, and the
sharper version came from the observation that at least one reader already applies the correct rule.
A correct implementation in each reader is the problem, not the fix. Every reader that gets it
right is one more place the rule exists only implicitly, and implicit agreement between independent
implementations decays silently — nothing fails at the moment one of them diverges. The PR page and a
CLI gate resolving the same rollup differently is a defect in the data model, not in either reader.
Revised:
data is read — not merely implemented correctly by each consumer. Latest conclusive run per
name decides;
SKIPPED/NEUTRAL/CANCELLEDare not verdicts; a name with any in-flight run ispending and cannot be resolved by an earlier terminal state. A reader that cannot see the
rule will invent one, and the two will agree until they do not.
The failure this prevents is concrete and already happened on #15864:
code-qualitycarriedCOMPLETED/SKIPPEDandIN_PROGRESSsimultaneously, so "did it pass?" had two defensible answers onone head — and the PR was simultaneously
mergeable=MERGEABLEwith three shards red, becausepython-suiteis not a required context (#10691). Every gate reported green on a PR that was not.mrveiss commentedon Sep 25, 2026
Evidence from a sweep of seven open PRs today, all on one author, while CI was being serialized so one PR at a time held the runners.
Three showed FAILURE in the status rollup. Every failing job's cause was a cancellation, not a defect:
##[error]lineverify-generated-typesgenerated-types verification cancelledmigration-matrixmigration matrix cancelledverify-generated-typesgenerated-types verification cancelledsmoke-testPath detection did not succeed (changes job: cancelled, docker filter: <unset>)Base was green at the time (
mainfully green on its latest SHA), so there was no shared upstream break to find.Why this costs more than a re-run
The gates are behaving correctly — a cancelled dependency is not a pass, and refusing to treat it as one is the same principle #12823 and #15995 are about. The cost is downstream of that: the rollup collapses "cancelled" into FAILURE, and the standing rule for an agent seeing a red PR is to root-cause it rather than re-run it. So each cancelled run buys an investigation that ends in "nothing was wrong", and the histogram gives no way to tell the two apart before opening the logs. I spent four API calls establishing it here, and the next session will spend them again.
The distinguishing information exists and is thrown away:
conclusion: CANCELLEDis present on the dependency job, and the dependent job's own conclusion is FAILURE. Anything that summarises PR readiness could separate "failed" from "could not run" from "never reported" — three states, currently two — and say which of them a re-run would fix.Not filing this separately: it is the same fan-out-and-cancel dynamic this issue describes, observed from the reader's end rather than the queue's.