ci: a run that never gets a runner is reported as failure, indistinguishable from a real test failure

What

A workflow run that never obtains a runner is reported to the PR as failure — the same signal a real test failure produces. Nothing distinguishes "your code is broken" from "no runner was available", so the standing rule red CI never merges fires on an infrastructure condition, and whoever reads it goes hunting for a defect in their diff that does not exist.

Evidence

Hardened Smoke Test, run 32984154162, on a PR branch:

field	value
`created_at`	`2026-08-26T15:07:57Z`
`updated_at`	`2026-08-26T15:22:39Z`
`status` / `conclusion`	`completed` / `failure`
its only job	`status: queued`, zero steps executed
annotations	none

A job with zero steps and no annotations did not fail a test — it was never dispatched. Fourteen minutes and forty-two seconds after creation the run was closed as failure.

The same workflow on a different branch at 15:22:42Z recorded startup_failure instead, so the outcome for the identical condition is not even stable.

Queue state measured at 15:49:19Z:

metric	value
queued runs, repo-wide	157
in-progress runs	2
self-hosted runners	2, both `online`, both `busy=false`
oldest queued run	`15:10:02Z` — 39 minutes waiting

Runners idle while 157 runs queue means the queued jobs are not waiting on the self-hosted pool.

Why it matters

The previous commit on the same branch passed this exact workflow (success). Only a test file changed between them. The failure was therefore guaranteed to be misread as a regression, and was: the delta was inspected, the job log fetched, the run object queried, and the steps enumerated before the zero-steps/queued shape made it obvious. That is the cost, per occurrence, multiplied by every PR in the backlog.

It also corrupts the merge gate in the safe direction only by luck — a dispatch failure blocks a good PR today, but the same indistinguishability means nobody can tell a genuinely red required context from a starved one without opening the run object by hand.

Acceptance criteria

A run that ends without its job executing a single step is reported distinguishably from a test failure — a neutral/cancelled conclusion, a named annotation on the check, or an automatic re-queue. Not a failure with an empty job.
Establish why 157 runs queue while both self-hosted runners sit online, busy=false — whether the queued jobs target hosted runners, an account concurrency cap, or a dispatch gate — and record the finding either way.
If the answer is a concurrency cap, the queue depth at which runs start timing out is written down somewhere a reader finds before assuming their diff is red.
CI Dispatch Watchdog / PR Queue Gate are checked for whether either already intends to cover this; if one does, fix it rather than adding a third mechanism.

Found while delivering bug(tests): api.terminal.router enumerates zero routes under python-suite shard 12/12 (unidentified sys.modules pollution) #15087 / PR test(terminal): fail the terminal route sweep when it enumerates nothing (#15087) #15113, where it cost a full root-cause pass on a clean diff.

Same class, second facet: `cancelled` also reads as `fail`

Hit while delivering PR #15113. The check No open blocks-merge issues reference this PR surfaced as a failure. It was not one:

field	value
check run `98236533493`	`status: completed`, `conclusion: cancelled`
`output.title` / `output.summary`	`null` / `null`
cause	superseded — a newer commit was pushed to the branch

Verified independently that the gate had nothing to say: the only two open blocks-merge issues are #14812 and #14677, and neither mentions #15113. A fresh run of the same gate is queued on the new head.

The cancellation itself is correct behaviour — concurrency superseding an obsolete run is what should happen. The cost is in the reporting: gh pr checks buckets cancelled under fail, so a superseded run and a genuinely red gate are the same string to anything reading that output, including an agent following red CI never merges. That is the same failure this issue is about — a non-failure presented as a failure — arriving by a second route.

Worth folding into the first acceptance criterion: whatever distinguishes a starved run should also distinguish a superseded one, or the guidance for reading check status should say explicitly that fail in gh pr checks is not the same as conclusion: failure and must be confirmed against the run object.

A third non-code cause producing an identical red, observed today

This issue records that a run which never gets a runner is reported as failure, indistinguishable from a real test failure. Adding a second, distinct cause with the same symptom, because the set matters more than the instance.

Observed on PR #15155, python-suite shard 12/12 (job 98783580325). The check reads failure. The job got a runner and executed 9 of 14 steps, so it is not the starved class this issue describes. What actually happened:

apt attempt 1/3 failed (exit 124) — retrying
apt attempt 2/3 failed (exit 124) — retrying
apt attempt 3/3 failed (exit 124) — retrying
##[error]apt did not succeed after 3 attempts — toolchain provisioning failed

Step 4 (Run ./.github/actions/setup-python-suite) failed on an apt-get timeout — exit 124 three times. Step 9 (Run unit tests — slm-backend) then failed downstream. No test result was produced at all.

So there are now at least three causes that present as the same red tile:

Cause	Steps executed	Distinguishable how
Never got a runner (this issue)	0	job has no executed steps
Toolchain provisioning failed	some, ending at the setup action	failing step is the setup action; log carries the `apt` retry text
A real test failure	most	failing step is a test step

The first two are re-queueable; the third must never be. They are currently told apart only by pulling GET /actions/jobs/{id} and reading the step list — which is a step most readers, and most tooling, will not take.

A concrete way this bites. A merge gate written against GET /commits/{sha}/check-runs cannot make this distinction at all, because that endpoint carries no steps array — only the jobs endpoint does. A gate that reads steps from the list response finds nothing every time and, depending on which way it defaults, either treats every red as infrastructure (and re-queues real failures) or every infrastructure blip as a real failure. I hit the first of those today and had to fix it: the correct default is that an undeterminable cause is not infrastructure, because mislabelling a real failure as a flake is the costly direction.

Worth folding into this issue's acceptance criteria: whatever reports the distinction should report which of the three it is, not merely "not a real failure".

fix(ci): say the diff is not indicted, not that re-running will fix it (

feat(ci): name which of five conditions produced a red check (#15139) (…

A concrete mechanism: re-dispatch instead of concluding

The evidence above already contains the detection fingerprint. A never-dispatched run is
distinguishable from a real failure without any guesswork:

Signal	Real failure	Never dispatched
run `conclusion`	`failure`	`failure` or `startup_failure` (this issue records both for the identical condition)
job `status`	`completed`	`queued`
steps executed	>= 1	0
annotations	>= 1	none

GitHub will not let a concluded run un-conclude, so "report it as pending" is not available.
Re-dispatching is: POST /actions/runs/{id}/rerun on a run matching that fingerprint turns an
infrastructure conclusion back into a pending check, which is the behaviour the required-context
contract actually wants. Red stays reserved for runs that executed something.

Where it belongs: ci-dispatch-watchdog.yml. That workflow already sweeps every open PR head
on a schedule and already owns this problem's mirror image — #12823 (runs created but parked
action_required) and #13045 (runs created but never allocating a job, context never reporting).
Both are "CI did not happen"; this is the third member of that family, differing only in that the
run closes itself as red instead of hanging. It should not become a fourth workflow.

Guards it needs, so a re-dispatch loop cannot mask a real problem:

cap re-runs per (workflow, head sha) — a run that never dispatches twice is a capacity or
configuration problem, and the second failure should stand and be visible
never re-run a fork PR run (the existing FORK SAFETY reasoning in that file)
publish the re-dispatch count on the watchdog's own commit status, so "this check is green
because it was retried" is legible rather than silent

This is not owner-blocked. It was batched with the owner-decision set on 31 Aug, but unlike
#15107 (a branch-protection contract decision) this one is implementable as it stands. Only #15107
needs an owner.

Related: the fork-safety constraint is about to change

ci-dispatch-watchdog.yml refuses to approve fork runs because "this repository ... runs
pull_request jobs on the self-hosted runner, so approving a fork run would execute
contributor-supplied code on the owner's machine".

PR #15394 moves the last 7 self-hosted-pinned jobs to ubuntu-latest (#15395). Once it lands, no
pull_request job targets a self-hosted runner and that specific reasoning no longer applies —
the #12823 parking this workflow exists to work around may become approvable rather than merely
reportable. Worth re-reading that block against the merged state rather than assuming either way.

The re-dispatch must be admission-controlled, or it becomes the problem it fixes

Adding the limits the mechanism above has to respect. A watchdog that re-runs eagerly re-queues
into whatever was already too full to dispatch -- the same speculative-work failure as the rest of
this pipeline, with a retry loop attached.

The limits that bind here:

Limit	Value	Consequence for the retry
GitHub-hosted concurrent jobs	~20 (free tier, public repo)	retrying while saturated just lengthens the queue
self-hosted runners	2	four per-PR-required workflows target them; see #15395
`GITHUB_TOKEN` REST rate limit	1,000/hr per repository inside Actions -- not the 5,000/hr a PAT gets	a per-PR sweep that lists runs and jobs burns this fast
Search API	30/min	a sweep must not reach for search endpoints at all
re-run endpoint	per-run	needs its own per-(workflow, sha) cap

Admission control, before any re-run:

Read queue depth and in-progress count first. If in-progress is at the ceiling, do nothing this
sweep -- a re-run cannot dispatch and only adds to the backlog.
Budget re-runs per sweep (a small fixed number), not per-PR. Unbounded fan-out over open PRs is
how a watchdog turns one outage into a rate-limit exhaustion.
Check X-RateLimit-Remaining and stop early with a published status rather than failing
mid-sweep -- a watchdog that dies halfway leaves exactly the ambiguous state it exists to
remove.
Prefer REST list endpoints with per_page=100 over per-run lookups; one listing answers for
many runs.
Cap per (workflow, head sha) as already noted -- a second non-dispatch is a capacity or
configuration signal and should stand, visibly, rather than be retried away.

The ordering matters: fix the dispatch pressure first, then add the retry. With #15394 landed
the self-hosted queue drains into idle hosted capacity, and the non-dispatch condition this issue
is about becomes rare enough that a small, budgeted retry is sufficient. Adding the retry while
the pressure is still there would mostly be retrying into a wall.

fix(ci): keep the watchdog under its size ratchet, repin the guard (#…

fix(test): fit the repinned guard inside its size ceiling (#15139)

fix(ci): re-dispatch infrastructure-caused reds, and close the merge_…

PR #15396 (merged to Dev_new_gui) references this issue with a close keyword.

fix(ci): re-dispatch infrastructure-caused reds, and close the #15394 merge_group gaps (#15139)

If this issue is fully resolved, close it manually. If work remains, no action is needed.

AC1 and AC4 delivered — `dfa525dc19` (PR #15396). AC2 partial, AC3 outstanding. Issue stays open.

AC1 — a non-dispatched run is now distinguishable. Verified in merged Dev_new_gui:
pipeline-scripts/ci_dispatch_retry.py present, and ci-dispatch-watchdog.yml invokes it. The
route chosen is the automatic re-queue the AC lists third: GitHub will not let a concluded run
un-conclude, so neutral was unavailable, and a re-dispatch returns the check to pending. Red goes
back to meaning "something executed and failed".

AC4 — checked, and fixed in place rather than added beside. ci_dispatch_watchdog.yml already
owned this problem's mirror image: #12823 (runs parked action_required) and #13045 (runs created
that never allocate a job). This is the third member of that family, differing only in that the run
closes itself red instead of hanging. ci_red_cause.py already classified the cause and re-queued
nothing; the missing half was acting on it. No third mechanism.

AC2 — measured today, and the answer is not a single cause

Both of this issue's readings are correct at their own moment, which is itself the finding:

when	queued	in progress	self-hosted runners
this issue, 26 Aug	157	2	online, `busy=false`
~14:00 today	213	17	both `busy=true`
~17:00 today	89	6-10 runs / 7 jobs	both `busy=true`

Three contributors separated:

Fan-out, the dominant one. CI/CD: cut the fixed per-PR toll — 63 checks, 117 runner-min, 20 min wall clock #15357 measures 63 checks and 117 runner-minutes per pull
request, and median queue delay 0s — capacity is not normally the constraint, volume is.
Dependabot. 9 open PRs regenerating ~50 queued runs; 126 were cancelled across two sweeps
today and they came back. Largest single block, ahead of every human branch combined.
The self-hosted pool, but only while four per-PR-required workflows targeted two machines.
At 17:00 the four deepest queues were exactly AutoBot CI/CD Pipeline, Code Quality,
Enforce Pre-commit Hooks and Frontend Testing Suite — 29 of 89. PR ci: make the required contexts reportable — merge_group triggers (#14054) + off self-hosted (#15395) #15394 removed that
contributor entirely; no job targets self-hosted any more.

So on 26 Aug the runners were genuinely idle and irrelevant; today they were saturated and central.
Two machines do not degrade gracefully — they are either unused or the whole bottleneck.

A caution this issue's own evidence supports: a queue-depth snapshot is not a wedge. #15185
recorded, then retracted, exactly that inference after watching 44 runs complete in 25 minutes. Run
counts also mislead — 6 in-progress runs was 7 in-progress jobs, and the ~20 free-tier ceiling
is on jobs.

AC3 — still open, and it needs a number nobody has

The queue depth at which runs start timing out is not established. It cannot be read off the
observations above: the two runs this issue caught (failure at 14m42s, and startup_failure for
the identical condition) came from different depths, and the outcome was not even stable between
them. Establishing it needs the re-dispatch shipped in AC1 to log depth-at-decision over a period —
which it now can, since it reads in-progress count and rate-limit budget before every decision.

Remaining on this issue: AC2's write-up is above but AC3's threshold is not established. Not
closing.

test(ci): install and remove the loaded modules in the same try/final…

fix(ci): sweep open PR heads for the infrastructure-retry watchdog (#…

PR #15399 (merged to Dev_new_gui) references this issue with a close keyword.

fix(ci): sweep open PR heads for the infrastructure-retry watchdog (#15139)

If this issue is fully resolved, close it manually. If work remains, no action is needed.

Closing — the mechanism now reaches the case this issue is about. 029f456de7 (PR #15399), on top of dfa525dc19 (PR #15396), verified against merged Dev_new_gui.

#15396 built the re-dispatch machinery but it could not act on an open PR's red check. Post-merge review established why: the step is skipped on pull_request — the only event where a PR head sha resolves — and on every other event HEAD_SHA fell back to github.sha, the base branch tip. Meanwhile run(sha, dry_run) classified exactly one sha with no open_pull_requests reference.

So a PR reddened by an empty runner pool stayed red while the sweep re-dispatched the base branch's own already-merged tip.

On merged base now:

ci_dispatch_retry.py                    open_pull_requests present
retry step env                          WATCHDOG_BASE_BRANCH, no HEAD_SHA
repo_tests/ci_dispatch_retry_test.py    present

The three surviving HEAD_SHA references belong to the read-only ci_red_cause.py step, which is correct — that one reports on the firing PR's head and never writes back, which is precisely why it can run before the self-modification guard.

The admission control survived the widening, which was the real risk. Going from one sha to N open PRs is the natural place to make the budget per-PR; it is not. redispatch() walks a single flattened candidate list built across every open PR with one done counter and a break, so exhausting on the first PR's reds stops the sweep before the second is touched. Per-PR budgets are how a watchdog turns one outage into rate-limit exhaustion.

All five gates are pinned by tests that did not exist before, each proven by mutation rather than asserted after the fact:

if done >= max_reruns  -> if False         2 reruns fired, not 1
count >= ceiling       -> count > ceiling  proceeded at ceiling
rate-limit -1          -> 999999           treated as headroom
attempt >= max         -> attempt >        attempt-2 run re-dispatched
collect_heads(...)     -> []               nothing re-dispatched

That last mutation is what proves the fix: with the open-PR sweep removed, nothing is re-dispatched at all — which was the merged behaviour, demonstrated rather than argued.

One irony worth recording. CI reported python-suite shard 5/12 as FAILURE on this PR. Both pytest runs passed — 2293 and 161 — and a sys.modules leak guard failed after them, because my new test file left three modules installed. A green test run presented as a test failure, in the PR that fixes exactly that reporting ambiguity. Fixed by install-and-remove in the same try/finally rather than by baselining (the guard's own instruction: it only shrinks), and verified with a probe test reporting LEAKED=none.

Confirming the closure holds on AC3, and recording the defect I shipped

Closing-comment review from the other side of this work.

AC3 does not block the closure, and the reason is worth writing down so nobody reopens it
later. It is conditional:

If the answer is a concurrency cap, the queue depth at which runs start timing out is
written down somewhere a reader finds before assuming their diff is red.

The established answer was not a concurrency cap. Measured across 31 Aug: fan-out (#15357 — 63
checks and 117 runner-minutes per PR, median queue delay 0s), dependabot volume (9 PRs
regenerating ~50 queued runs; 126 cancelled across two sweeps and they returned), and — until PR
#15394 — four per-PR-required workflows targeting two self-hosted machines, 29 of 89 queued
runs at 17:00 while hosted capacity ran 7 jobs against a ~20 ceiling. Hosted concurrency was never
observed as the binding constraint. The precondition never fires, so AC3 is vacuous rather than
outstanding. My earlier comment listed it as unmet without noting it was conditional.

And the defect in dfa525dc19 was mine. I wired the retry step if: github.event_name != 'pull_request' — reasoning that PR events run PR-head code and the sweep is scoped to the firing
PR — without following through to what HEAD_SHA then resolves to. On every remaining event it is
github.sha, the base tip. So the machinery re-dispatched the base branch's already-merged head
and could never touch the open PR reds it was built for. The feature shipped unable to do its job,
and my PR body claimed only that the re-dispatch path "needs a real non-dispatched run to act on" —
which read as untested, not unreachable.

029f456de7 (#15399) is the fix, and it is better evidenced than the original: five gates pinned
by mutation rather than assertion, including collect_heads(...) -> [] proving that without the
open-PR sweep nothing is re-dispatched at all — i.e. demonstrating the merged behaviour rather than
arguing about it.

Two things I take from this for anyone picking up adjacent work:

An if: that excludes an event also silently changes what every ${{ }} in that step resolves
to. The event guard and the sha expression have to be read together.
"Cannot be verified before merge" is a claim that needs testing itself. Here it was false: the
unreachability was provable statically, and a mutation test found it in one pass.

Navigation Menu

Repository navigation

ci: a run that never gets a runner is reported as `failure`, indistinguishable from a real test failure #15139

What

Evidence

Why it matters

Acceptance criteria

Related

Same class, second facet: `cancelled` also reads as `fail`

A third non-code cause producing an identical red, observed today

A concrete mechanism: re-dispatch instead of concluding

Related: the fork-safety constraint is about to change

The re-dispatch must be admission-controlled, or it becomes the problem it fixes

AC1 and AC4 delivered — `dfa525dc19` (PR #15396). AC2 partial, AC3 outstanding. Issue stays open.

AC2 — measured today, and the answer is not a single cause

AC3 — still open, and it needs a number nobody has

Confirming the closure holds on AC3, and recording the defect I shipped

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

ci: a run that never gets a runner is reported as failure, indistinguishable from a real test failure #15139

Description

What

Evidence

Why it matters

Acceptance criteria

Related

Activity

mrveiss commented on Aug 26, 2026

Same class, second facet: cancelled also reads as fail

mrveiss commented on Aug 28, 2026

A third non-code cause producing an identical red, observed today

mrveiss commented on Aug 31, 2026

A concrete mechanism: re-dispatch instead of concluding

Related: the fork-safety constraint is about to change

mrveiss commented on Aug 31, 2026

The re-dispatch must be admission-controlled, or it becomes the problem it fixes

github-actions commented on Aug 31, 2026

mrveiss commented on Aug 31, 2026

AC1 and AC4 delivered — dfa525dc19 (PR #15396). AC2 partial, AC3 outstanding. Issue stays open.

AC2 — measured today, and the answer is not a single cause

AC3 — still open, and it needs a number nobody has

github-actions commented on Aug 31, 2026

mrveiss commented on Aug 31, 2026

mrveiss commented on Sep 1, 2026

Confirming the closure holds on AC3, and recording the defect I shipped

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

ci: a run that never gets a runner is reported as `failure`, indistinguishable from a real test failure #15139

Same class, second facet: `cancelled` also reads as `fail`

AC1 and AC4 delivered — `dfa525dc19` (PR #15396). AC2 partial, AC3 outstanding. Issue stays open.