Repository navigation
Auto-update starves long-running PRs: every merge restarts their CI, so the busiest nights are the ones nothing old lands #16015
Description
Activity
The rebase is not required by any gate — measured
"This branch is out-of-date with the base branch"is what a reader sees on these PRs, and it is easy to read as the reason they cannot merge. It is not. Checked all three places a requirement could live:where setting classic branch protection required_status_checks.strict= falseMainrulesetrules are deletion,non_fast_forward— no status-check rule at allRequire review for external contributorsrulesetpull_requestonlyNothing in this repository requires a branch to be up to date before merging. The banner is informational, with an "Update branch" button beside it, and GitHub will merge a behind-by-six PR without complaint.
What that changes
It does not make auto-update pointless. Being current still matters for a different reason: a green run describes the merge base it ran against, and base may have moved in a way that breaks the combination. That is a correctness argument and it stands.
But it removes the argument for updating a branch while its suite is running. The PR was not going to merge mid-suite regardless of behind-ness, so the rebase gains nothing at that moment and costs the entire run — and with
cancel-in-progress, costs it twice: the in-flight run is killed and restarted.So the first acceptance criterion above is cheaper than it looked. "Do not re-push a branch whose suite is in flight" defers the update by at most one suite length and forfeits no gate, because there is no gate to forfeit.
The reading error worth recording
The banner and a blocking requirement produce the same sentence on the same page, and only one of them stops a merge. Everyone on this board — me included — has been treating behind-ness as a merge blocker on this repo for hours, on the strength of a UI string, without checking whether
strictwas set.That is this repository's recurring defect in a new place: a label read as a verdict. Same as the shard number that is not a test identity, the
authorfield that is not a session identity, andmergeStateStatus=UNKNOWNthat is not a statement about mergeability. The instrument said something narrower than what it was read to mean.Immediate mitigation available without a code change
Base moving is what triggers the churn, and base moves when we merge. Holding merges for one suite length (~40 min) lets #15972 and #15977 — which have no failing checks, only pending ones — complete a suite for the first time in 16 hours. #15978 and #15980 have real guard failures and must not merge regardless.
The measurement, from the PRs themselves
Each auto-update merge is a new head SHA and therefore a full suite restart. Counting them per PR gives the reset count directly:
PR auto-update merges total commits real content #15972 9 10 1 #15977 8 9 1 #15978 7 8 1 #15980 4 6 2 28 restarts across four PRs, each costing ~24 checks — roughly 670 check runs to produce zero completed suites. Every one of those PRs is nine-tenths bookkeeping by commit count.
This is the second acceptance criterion above, and it turns out to need no new instrumentation:
gh api pulls/N/commitsalready carries it. A PR whose commits are overwhelminglyMerge branch 'Dev_new_gui' into ...is a PR that has been restarted more often than it has been worked on.The diagnosis is starved too
Both starved PRs currently report
verify-generated-typesas never-reported, and their heads are 8 minutes old — rebased again while this was being written.That number cannot be interpreted.
never-reportedis ambiguous between "has not started yet" and "will never start", which is the exact ambiguity #15995 documents, and an 8-minute-old head is well inside the window where the benign reading is plausible.So: nobody has ever observed these PRs in a steady state. Every reading is taken minutes after a reset, which makes the benign and the pathological explanations indistinguishable. The starvation does not merely block the merge — it blocks the diagnosis of the block. Any fix here has to create one uninterrupted window first, or every measurement taken to justify the fix will be taken during a restart.
That is worth stating plainly because it inverts the usual order: normally you diagnose, then fix. Here the first fix has to be "stop resetting it", and only then does the instrument say anything.
Correction: one of the four was never starved, and I grouped it wrongly
A 25-minute window with no merges was enough to separate them:
PR head age suite state #15972 14 min 21 pass, 14 pending — progressing #15977 14 min 21 pass, 10 pending — progressing #15978 15 min 23 pass, 23 pending — progressing #15980 371 min 2 fail, 53 pass, 0 pending — completed #15980 finished its suite six hours ago. It has a verdict — the protobuf tripwire in
requirements_pin_parity_test.py— and it has been sitting on it, unread, ever since. It was not starved by anything. It was diagnosed and ignored.I put all four in the same bucket because they shared the visible symptom, open 16 hours and not merged. That is the identical error this issue is about, committed while writing it up: a shared symptom read as a shared cause. Same shape as the shard number, the
authorfield, and the out-of-date banner — a label doing less work than the reading it was given.The distinction matters for the fix, because the two groups need opposite things:
- Starved (chore(deps): bump the all-dependencies group in /autobot-backend with 16 updates #15972, chore(deps): bump the all-dependencies group in /autobot-infrastructure/shared/docker/ai-stack with 12 updates #15977, chore(deps): bump the all-dependencies group in /autobot-tts-worker with 4 updates #15978): they need an uninterrupted window. Nothing else will help; no amount of attention produces a verdict from a suite that keeps restarting.
- Stalled on a verdict (chore(deps): bump the all-dependencies group across 1 directory with 22 updates #15980): it needs somebody to read the answer it already produced. A window changes nothing for it.
And a sweep filtering on "failing" or "green" misses both groups — the starved ones because they are permanently running, the stalled one because
python-suiteis not a required context, so it reports GREEN-BUT-OTHERS-FAILING and slides past any check keyed on the required set.Consequence for the acceptance criteria
The fourth criterion above asked the measurement to distinguish "never started" from "restarted repeatedly". It needs a third: "completed, verdict unread". That state is not pending and not blocked, and it is the one that looks most like progress while being the most inert.
The bottleneck measured, and it is not where the runner fleet suggests
A 58-minute merge-free window let me measure the thing instead of inferring it.
Queue depth: 52 workflow runs queued.
11 AutoBot CI/CD Pipeline 6 Security Scanning 5 Verify Generated Types 4 Migration Gate 3 API Wiring Audit 3 Code Quality 3 Frontend Required ContextSelf-hosted runners: 2 online, and BOTH IDLE.
Little-Slave status=online busy=false labels=self-hosted,Linux,X64 Second-Little-Slave status=online busy=false labels=self-hosted,Linux,X64So the 52 queued runs are not waiting on the self-hosted fleet — that fleet has spare capacity right now. They are waiting on GitHub-hosted concurrency, because the jobs that matter (
python-suite's 12 shards,verify-generated-types,code-quality) declareruns-on: ubuntu-latest.I nearly reported "2 runners against 52 queued runs" as the cause. It reads that way, the numbers sit next to each other, and it is wrong: an idle runner is not a bottleneck. Adjacent numbers are not a ratio — the same error as the co-located log text, one tick later.
What this does to the cost estimate
It makes the waste worse, not incidental.
Over the window the three starved PRs moved from 17 pending to 11 pending in 22 minutes — roughly one check every 3.7 minutes per PR, with all three contending. Extrapolated, a full suite here is ~100 minutes of wall clock, almost all of it queueing rather than running.
That reframes the restart arithmetic. 28 auto-update restarts × ~24 checks ≈ 670 check runs, and every one of them was drawn from the same saturated GitHub-hosted pool that the PRs which can finish are queued in. The restarts do not merely fail to help the starved PRs — they lengthen the queue for everything else, including the young PRs that merge quickly and thereby trigger the next round of restarts.
And it explains why the loop is stable rather than self-correcting. A ~100-minute suite cannot survive a base that moves every 20–30 minutes. The restart interval was never within a factor of two of the completion time, so no amount of patience was ever going to clear these.
Sharpened acceptance criteria
- The first criterion's threshold is ~100 minutes, not "a suite length" loosely — a fix that defers a rebase by 40 minutes changes nothing here
- Wasted capacity is reported in check runs consumed against completed suites, since the pool is shared: a restart is a cost paid by every other PR in the queue, not only by the one restarted
- The self-hosted fleet's idleness is either explained or used — two idle runners beside a 52-deep queue is either a labelling gap worth closing or a fact worth writing down, and right now it is neither
Measured: one merge pair cost eight PRs their in-flight suites
I merged #16018 and #15972 together — deliberately batched as one base move, exactly as this issue recommends. Board state a few minutes later:
PR before after #15977 pass=57 pending=1 pass=2 pending=30 #15978 fail=2 pass=53 pass=2 pending=27 #15996 pass=44 pending=8 pass=3 pending=26 #16012 fail=1 pass=49 pending=1 pass=2 pending=26 #16027 pass=17 pending=21 pass=3 pending=28 #16047 pending=4 pass=6 pending=21 #16048 pass=1 pending=24 pass=7 pending=21 #16049 pass=2 pending=27 pass=10 pending=21 #15977 is the one that hurts. It was 57 passed with a single check outstanding — minutes from a verdict after sixteen hours of never completing one. Its head moved to a new SHA between two of my own commands, and it is now at 2.
What this confirms, and what it does not
Confirms: the loop is real, and batching is necessary but not sufficient. Two merges in one window cost one restart rather than two — that part works. But one restart still resets every open PR, so on a board of nine the cost of any merge is eight suites, and the closest-to-done PR loses the most.
Does not confirm: that batching helps enough. I recommended it here on the reasoning that four merges inside ten minutes destroy one suite instead of four. That is true and it is not sufficient: a nine-PR board with a ~100-minute suite and any merge cadence at all leaves the slowest PRs unable to finish. Batching reduces the frequency of the reset, not its blast radius.
The correction this forces to the mitigation
Earlier I proposed "hold merges for one suite length" as the cheap fix, and it worked — a 25-minute window let three starved PRs progress, and a 58-minute one let #15972 and #15977 complete for the first time in sixteen hours.
But that mitigation costs the merge queue, and it only works while someone is deliberately not merging. It is a manual workaround for an automated loop, and it competes directly with the instruction to merge green PRs promptly. The two cannot both be followed on a busy board.
So the fix has to be in the auto-update trigger, not in merge discipline. The first acceptance criterion above — do not re-push a branch whose suite is in flight — is the one that removes the conflict, because it lets merges proceed at any cadence without resetting anything mid-run.
Evidence for the "nothing requires this" half
Worth restating beside the numbers:
required_status_checks.strictis false, and neither ruleset carries a status-check rule. The rebase that destroys these suites is not buying a merge gate — I merged #16005 five commits behind base with no complaint, and #16023 two behind.Measured twice today, ~10 hours apart. The starvation is real and worsening, but the mechanism in the title is not the dominant cost — and the difference changes the fix.
this morning now (21:01–21:22Z) Window for 200 runs 60 min 21 min Still queued / in progress 12 (6%) 92 (46%) Cancelled 0 3 (1.5%) Failures 0 3 Runs per active branch ~28 ~27 Active branches 8 5 Arrival rate has roughly tripled while capacity has not, and 46% of runs no longer complete at all.
The refinement
This issue's premise is that
cancel-in-progresskills suites before they finish. That happens — 3 in 200 — but at 1.5% it is not what is stopping PRs from completing. The dominant cost is arrival rate exceeding runner capacity, and cancellation is a symptom of the same overlap rather than its cause.The distinction matters because the fixes diverge:
- If the cost were cancellation, the fix is concurrency-group scoping — let a superseded run die, keep the newest.
- Since the cost is volume, that changes nothing: the newest run still queues behind 90 others. The fix has to reduce runs created, not runs killed.
Where the volume comes from
Each open PR carries ~27 workflow runs, and the full set re-fires on every push to its branch. The auto-update bot merges base into every open PR branch on every push to
Dev_new_gui, so one merge to base creates ~27 × (open PRs) runs. At 5 open PRs that is ~135 runs per merge; at the 8 open this morning it was ~220.That is quadratic in exactly the two things we are optimising for — merge often, and keep several PRs in flight. Every merge I make as watcher re-queues the entire board, so the gate's own throughput is self-limiting.
What I am not claiming
I have not measured runner count, per-workflow duration, or how many of the ~27 are genuinely required — several are path-filtered and skip cheaply, and a skipped run still occupies a queue slot but not a runner. So "reduce runs created" is the direction the data supports; which runs to cut needs the per-workflow breakdown I have not taken.
The two obvious candidates, both unmeasured by me: whether the auto-update bot needs to run on every base push rather than on demand before merge, and whether path filters could skip at the workflow level rather than inside jobs that still get scheduled.
Why it is worth acting on now
This is currently the binding constraint on the whole board. Five PRs are open with zero failures and zero unresolved review threads between them — every one is waiting purely on runners. Nothing is wrong with any of the work; the queue is the only thing stopping five merges.
Correction to my own comment above: the amplification mechanism I attributed this to is capped, and my per-merge figure was wrong by roughly 5×. The measurements stand; the cause does not.
I wrote that one merge to base makes the auto-update bot re-fire ~27 workflows on every open PR — ~135 runs at five PRs, quadratic in merges × open PRs. That is not what the workflow does.
.github/workflows/auto-update-pr-branches.yml::4 "refreshes AT MOST ONE open PR targeting Dev_new_gui: the oldest one that is behind" :29 "MAX_BRANCH_UPDATES already caps actual refreshes at one per push" :136 MAX_BRANCH_UPDATES: '1' :249 [ "$UPDATED" -lt "$MAX_BRANCH_UPDATES" ] || breakOne merge refreshes one PR. So a base merge costs ~27 runs, not ~135, and there is no quadratic term. The #14022 comment records that this cap exists precisely to stop reset amplification — the mechanism I described had already been fixed before I described it.
Checked against tonight's actual history, not just the config. Across every open branch, exactly one auto-update commit exists:
e51bc1af7f github-actions[bot] Merge branch 'Dev_new_gui' into issue-15956-hook-askOne refresh, on one branch — which is exactly what a cap of 1 predicts. The only other bot commit anywhere is
b1ea5663ce,Auto-fix: code formatting, from a different workflow.What survives and what changes
The starvation measurements stand — 200 runs in 21 minutes, 46% never completing, arrival rate tripled in ten hours. Those were counted, not inferred.
The conclusion survives: the lever is runs created, not runs killed.
What changes is whose runs. They are overwhelmingly ours. Six pushes tonight at ~29 runs each is ~174 against the 200 sampled — the bot contributed one refresh. My model predicted bot commits scattered across every branch; the history has one.
That redirects the fix:
- The formatting bot is the one real bot amplifier, and it does not merely add runs — its push supersedes the in-flight suite on that branch, so the runs it displaces were already paid for. Deleting it (ci(perf): delete the formatting bot — it is the third application of the same tools (#15934) #16081) is supported directly by this.
- If the runs are ours, the lever is pushing fewer times, not firing fewer workflows per push — local preflight (Local preflight must cover the 10 required checks #15933), so a push that would fail is not made.
The error worth recording
I reasoned from "the bot merges base into open PR branches" to "therefore into all of them" without reading the cap, then multiplied by the number of open PRs and reported the product. A plausible mechanism, multiplied by a real count, produces a false number with a true-looking magnitude — and it was in the same comment where I was careful to list what I had not measured. I had not measured this either; I just did not notice that I had not.
Measured bot-push count, replacing two estimates — both of them wrong, in opposite directions
Two sessions modelled the amplification and neither measured it. Here is the count, taken over each open PR's full history since its merge-base with
Dev_new_gui, not a fixed window:#15980 dependabot pip bot=5 of 16 commits 5 × "Merge branch 'Dev_new_gui' into ..." #15996 issue-15956-hook-ask bot=3 of 21 commits 3 × auto-update merge #16072 issue-16071-redis-unit-names bot=1 of 7 commits 1 × "Auto-fix: code formatting" #16081 issue-15934-drop-format-bot bot=0 of 2 #16082 dependabot uv bot=0 of 2 #16083 dependabot npm_and_yarn bot=0 of 2 #16084 issue-15987-silent-case bot=0 of 2Eight auto-update refreshes and one formatting push across seven branches.
What each estimate got wrong
"One merge re-fires ~27 workflows on every open PR — ~135 runs per merge." Wrong by 5×.
auto-update-pr-branches.yml:136setsMAX_BRANCH_UPDATES: '1', enforced at:249([ "$UPDATED" -lt "$MAX_BRANCH_UPDATES" ] || break) and documented at:4and:29. One merge refreshes one PR. There is no quadratic term."Zero auto-update commits anywhere; the bot contributed one." Wrong by 8×. That was
git log --oneline -3per branch reported as an absolute — a real observation over an unstated window, stated universally. The window was chosen for readable output and then quoted as coverage.What the measurement actually shows, which is neither
The cap prevents fan-out across PRs and does nothing about repetition on the same PR over time. #15980 alone absorbed five refreshes; #15996 three. Each restarts that PR's suite, discarding whatever had completed.
So the cost is not per-merge fan-out, and not negligible. It is long-lived PRs accumulating refreshes — which points at merging faster, or at the merge queue (#15937/#14054), rather than at the cap. The cap is working as designed and should not be touched.
The two levers this does support
#16081 — delete the formatting bot. It is the only amplifier that supersedes an in-flight suite rather than adding to it. Measured instance:
b1ea5663celanded onissue-16071-redis-unit-namesafter a push, superseded that suite, and removed one line — invalidating the file-size ceiling the superseded push had just set, so the audit then failed for the opposite reason.#15933 — local preflight. Against a ~200-run sample, six human pushes at ~29 runs each is ~174. The bot contributed nine commits total. The volume is overwhelmingly ours, so the lever is pushing fewer times, not firing fewer workflows per push.
Method note, since this issue is about measurement
Both wrong figures were produced inside comments that carefully listed what had not been measured. One listed runner count and per-workflow duration as unmeasured, then reported a modelled product as a fact. The other listed nothing, having mistaken a three-commit sample for a census.
The common failure is not carelessness about evidence — it is that the scrutiny went to the numbers each author already doubted, and the confident one got none. That is anti-correlated with where the error was: a plausible figure buys its own exemption from checking.
The distribution, which is the actual finding
The count above says eight auto-update refreshes across seven branches. Independently re-counted over
origin/Dev_new_gui..origin/<branch>— bounded by the divergence point rather than an arbitrary-N— the same nine commits, and this is how they fall:branch commits auto-update formatting dependabot pip 16 5 0 issue-15956-hook-ask 21 3 0 issue-16071-redis-unit-names 7 0 1 issue-15934-drop-format-bot 2 0 0 issue-15987-silent-case 2 0 0 The refreshes are not spread across branches. They are perfectly correlated with branch length. The two long-lived branches absorbed all eight; the three short ones absorbed zero.
So a branch's refresh count is a function of how long it stays open, not of how many PRs exist. That is why both earlier models failed: one multiplied by open-PR count, the other sampled a fixed window and missed the tail. The variable neither used was age.
What that makes the remedy
The lever is time-to-merge. A PR merged within the hour takes zero refreshes; one open all evening takes five, each restarting its suite and discarding whatever had completed.
This is not an argument for changing
MAX_BRANCH_UPDATES— the cap is doing exactly its job, and #14022 records what removing it cost. It is not primarily an argument about the formatting bot either, which accounts for one of the nine. It is an argument for:- the merge queue (Enable the merge queue — no PR merges against a stale base #15937 / ci: migrate main to native GitHub merge queue (requires merge_group triggers first) #14054) — a queue tests the merged result, so branches do not need dragging forward at all, and the eight refreshes exist only because
strict: falserequires manual freshening; - not sitting on green PRs — whoever holds the merge gate directly controls this variable, and holding a green PR for an hour costs a full suite re-run rather than nothing.
Method note
Three figures were produced for this: ~135 per merge, "one, anywhere", and the measured nine. The first modelled a mechanism with no sample; the second sampled with an unstated bound. Both were stated in comments that carefully enumerated other unmeasured quantities.
The generalisable part is that each author's scrutiny went to the numbers they already doubted, and the number they were confident about got none — which is anti-correlated with where the error was, because confidence is what buys the exemption from checking. Two independent authors, same shape, one evening.
- the merge queue (Enable the merge queue — no PR merges against a stale base #15937 / ci: migrate main to native GitHub merge queue (requires merge_group triggers first) #14054) — a queue tests the merged result, so branches do not need dragging forward at all, and the eight refreshes exist only because
Cross-link: #17760 (filed tonight) — the dependabot half of this mechanism: 11 grouped dependabot PRs × ~20 workflows re-triggered on every base move by the auto-update bot; the fix there excludes
dependabot/*heads fromauto-update-pr-branches.ymland groups across ecosystems. Same file as #17743's selection defects — one workflow PR.
The loop
Four dependabot PRs — #15972, #15977, #15978, #15980 — have been open 16 hours and have never completed a CI suite. Younger PRs opened tonight went green and merged within the hour.
The mechanism is a loop, not a queue:
auto-update-pr-branchesmerges base into every open PR branch whenever base moves, not on a timer.python-suiteshards.verify-generated-types.ymland its siblings setcancel-in-progress: ${{ github.event_name == 'pull_request' }}, so the in-flight run is killed, not allowed to finish.Evidence — the last three commits on #15977's branch:
A merge, then a base commit, then another merge. Its current head is minutes old and its suite reads 23 running, 1 success — starting over for at least the seventh time.
Why it is a starvation loop and not slowness
The failure gets worse the better the fleet performs. Every merge resets every older PR. On a quiet night the old PRs would finish; on a productive one they cannot. Nothing in the system notices, because each individual restart looks like normal CI activity, and the PR's status page always shows checks moving.
Rough cost tonight: 6 base moves × 4 stalled PRs × ~24 checks ≈ 550 check runs consumed to produce zero completed suites. That is queue capacity taken from PRs that can finish.
Why nobody sees it
A PR stuck this way is never red and never green — it is permanently running. Every review sweep that filters on "failing" or "green" skips it, and a human glancing at it sees activity and assumes progress. It is invisible in exactly the way an un-started check is invisible to a status histogram, which is the shape #15995 was written about: a state that contributes to no bucket.
Fix direction
Not "run less CI". The options are about when a rebase is worth restarting a suite:
dependabot/**from auto-update. Dependabot rebases its own PRs on request, and these are the branches with the longest suites and the least urgency.Acceptance criteria
Related