ci(runners): retire the self-hosted runners — nothing in CI needs them, and three required contexts depend on a two-machine pool

Supersedes the direction taken in #15311 (which closed #15310). Not a silent undo — the
premise is being corrected, so it is recorded here.

What #15310 assumed

#15310 was filed on the observation that "the intent is to run CI on the self-hosted
runners" while 92 of 96 jobs targeted ubuntu-latest. That intent was never grounded in a
hardware requirement — the runners were online and idle, so the gap read as a defect.
#15311 moved the first batch across on that basis.

Re-examined, the premise does not hold. Nothing in CI needs a self-hosted runner.

Evidence

Seven jobs are pinned to self-hosted, all with generic labels — [self-hosted, Linux, X64].
No npu, gpu, cuda, or any hardware-specific label anywhere in .github/workflows/:

Workflow	Jobs	Publishes a required context?
`code-quality.yml`	`code-quality`	yes
`enforce-precommit.yml`	`verify-precommit-config`, `check-instructions`	yes
`frontend-test.yml`	`unit-tests`, `security-scan`, `build-test`, `test-summary`	yes (`Unit & Integration Tests`)

Three of the ten required contexts therefore depend on a two-machine pool. A runner
outage blocks every open PR in the repository. That is recorded history, not a hypothetical:

ci: strict branch protection deadlocks when the self-hosted runner is offline — the branch-updater is pinned to that same runner #13401 — strict branch protection deadlocked when the singleton went offline
ci: #13405's wedged-concurrency state is live again — 5 Frontend Testing Suite runs queued 39h, including Dev_new_gui, while both self-hosted runners are online #15185 — a queued predecessor holding a concurrency group while runners were unavailable
bug(ci): self-hosted jobs cannot start — codeload returns HTTP 429 fetching actions, and runner-health reports starvation #14444 — self-hosted jobs could not start at all (codeload HTTP 429)

Why hosted is strictly better here

No cost. The repository is public, so GitHub-hosted runners are free with ~20
concurrent jobs. Self-hosted buys no minutes and no capacity that matters — CI/CD: cut the fixed per-PR toll — 63 checks, 117 runner-min, 20 min wall clock #15357
measured median queue delay at 0s.
Contention manufactures red checks. test(backend): probe-concurrency test proves concurrency with a wall clock, and fails on runner contention #14157 (probe-concurrency test fails on runner
contention) and test(perf): process-offload heartbeat test flakes on a contended CI runner (produced 1 tick, needs 10) #15266 (a starved event loop produces too few heartbeat ticks to measure)
are load artifacts, not defects. Two runners carrying 180-minute code-quality and
marker-tests timeouts guarantee that load.
Toolchain maintenance. Self-hosted runners on Ubuntu 25.10 (non-LTS): actions/setup-python has no 3.14 build, blocking the self-hosted migration #15313/ci(runners): schedule the watchdog and fix setup-python on non-LTS self-hosted (#15309, #15313) #15314 exist only because actions/setup-python has no
build for the runners' non-LTS OS, so setup-python-ci branches on runner.environment.
That branch exists solely to serve these runners.
The host runs the live stack. CI load competes with the production services on the
same machine — the CRITICAL(ops): autobot-backend throttled by an out-of-band MemoryHigh drop-in — 21,052 reclaim events, STAT=D, health timing out, systemd reporting active #13765 throttling class.
Security. Self-hosted runners on a public repository are a documented anti-pattern:
a fork pull request can execute arbitrary code on the host. The fork-approval policy
mitigates it; retiring the runners removes the exposure entirely.

marker-tests.yml already makes this argument in its own header: "runs-on: ubuntu-latest on
every job. The self-hosted runner is a singleton."

Tasks

Move all 7 jobs back to ubuntu-latest
Retire self-hosted-runner-health.yml — it probes a pool that no longer exists
Remove the runner.environment branch from .github/actions/setup-python-ci (ci(runners): schedule the watchdog and fix setup-python on non-LTS self-hosted (#15309, #15313) #15314)
Review ci-dispatch-watchdog.yml for self-hosted assumptions
Deregister Little-Slave and Second-Little-Slave — owner action, last
Record the wall-clock delta for code-quality (documented at ~601s self-hosted; the
hosted comparison is unmeasured and is the one open question)

Acceptance criteria

No workflow job targets self-hosted
All ten required contexts run on GitHub-hosted runners
code-quality and Unit & Integration Tests durations measured before/after on 3 PRs
Runners deregistered, or a documented reason they are kept for non-CI use

CI runs entirely on GitHub-hosted runners while both self-hosted runners sit idle: 92 of 96 jobs target ubuntu-latest #15310 / ci(runners): move code-quality and the pre-commit gate onto the self-hosted runners (#15310) #15311 — the direction this supersedes
ci: strict branch protection deadlocks when the self-hosted runner is offline — the branch-updater is pinned to that same runner #13401, ci: #13405's wedged-concurrency state is live again — 5 Frontend Testing Suite runs queued 39h, including Dev_new_gui, while both self-hosted runners are online #15185, bug(ci): self-hosted jobs cannot start — codeload returns HTTP 429 fetching actions, and runner-health reports starvation #14444 — outages caused by the dependency
test(backend): probe-concurrency test proves concurrency with a wall clock, and fails on runner contention #14157, test(perf): process-offload heartbeat test flakes on a contended CI runner (produced 1 tick, needs 10) #15266 — tests that fail on contention rather than on defects
CI/CD: cut the fixed per-PR toll — 63 checks, 117 runner-min, 20 min wall clock #15357 — the per-PR cost umbrella; capacity is not the bottleneck

ci(runners): move the last 7 jobs off self-hosted onto ubuntu-latest (#…

Discovered while moving the jobs: a second dead branch, not in the task list above

.github/actions/free-disk-space/action.yml:44-61 carries a runner.environment == 'self-hosted'
Docker-reclaim step, the same shape as the setup-python-ci branch already listed.

Once PR #15394 lands, no job targets self-hosted, so runner.environment can only ever be
github-hosted and neither branch is reachable. Adding it here so it is not lost:

Remove the runner.environment == 'self-hosted' reclaim step from
.github/actions/free-disk-space/action.yml

Deliberately not removed in #15394. Both branches stay reachable-in-principle until the runners
are actually deregistered, and deleting them inside the PR that could be reverted would take the
capability with it. They become provably dead only once the deregistration in this issue is done —
that is the right moment for both.

Live measurement: the self-hosted pool is the current queue bottleneck

Measured just now, repo-wide:

Metric	Value
queued runs	89
in-progress (GitHub-hosted)	6-10, against a ~20 concurrent ceiling
queued runs on self-hosted workflows	29
self-hosted runners	2

Queued runs by workflow, top of the list:

9  AutoBot CI/CD Pipeline      (ci.yml)            self-hosted
8  Enforce Pre-commit Hooks    (enforce-precommit) self-hosted
7  Code Quality                (code-quality)      self-hosted
5  Frontend Testing Suite      (frontend-test)     self-hosted
5  Verify Generated Types                          hosted
3  Security Scanning                               hosted

The four deepest queues are all self-hosted-pinned workflows, and hosted capacity is running at
roughly a third of its ceiling. Checked for concurrency-group stacking as the alternative
explanation -- zero same-(workflow, ref) pairs queued, so that is not it.

This does not contradict #15139's measurement, it dates it. That issue recorded 157 queued
with both runners busy=false and concluded the queue was not waiting on the self-hosted pool.
Both readings are correct at their own moment: the pool is idle when nothing needs it and is the
hard stop when four per-PR-required workflows do. Two runners is not a capacity that varies
gracefully -- it is either unused or saturated.

PR #15394 moves all seven pinned jobs to ubuntu-latest, which drains this specific queue into
capacity that is currently two-thirds idle. That is a reason to land it ahead of the other open
PRs, independent of the merge-queue work it also carries.

ci(runners): merge_group triggers for every required context, and off…

fix(ci): re-dispatch infrastructure-caused reds, and close the merge_…

Live evidence for this, from the session that caused it

I filed #15310 and implemented #15311. This issue's re-examination is correct and mine was not. My premise was that idle runners plus 92 of 96 jobs on ubuntu-latest read as a defect — I never checked whether anything needed self-hosted hardware, and as this issue establishes, nothing does. Every pinned job uses generic [self-hosted, Linux, X64] labels.

The predicted failure then happened, on 2026-08-31, before I saw this issue. I filed it separately as #15407; it belongs here as evidence.

What occurred

Nine dependabot PRs arrived together. verify-precommit-config — required and non-path-filtered, so it runs on every PR — was cancelled after 1h19m:

concl=cancelled  started=2026-08-31T15:08:30Z  completed=2026-08-31T16:27:55Z

Its timeout-minutes is 30. The other ~50 minutes were queueing. Both runners were busy=true.

It then recurred on a markdown-only PR (#15413), which rules out anything about the change under test. The queue does not care what the PR contains.

The detail worth keeping

timeout-minutes bounds execution, not time-to-start. So a queue stall surfaces as a cancellation that looks identical to a slow job — and invites raising the bound, which cannot possibly help. That misreading cost me time here, and it is the failure mode anyone debugging this will hit first.

Where I disagree slightly, and where I do not

I do not defend #15311's direction. But two of the three jobs it moved are cheap and path-filtered:

Job	Frequency	Exposure
`verify-precommit-config`	every PR, required	the one that stalls
`code-quality`	path-filtered to backend	lower arrival rate
`check-instructions`	every PR, seconds long	releases its runner almost at once

If this issue's full retirement lands, that distinction is moot and the simpler outcome is better. If instead a partial position is chosen, verify-precommit-config is the one that must move back — it is the required, unfiltered, expensive one.

There is also a cheaper-job option: #15392 scopes enforce-precommit to the PR diff rather than --all-files, which would cut occupancy sharply. That reduces the symptom without addressing this issue's actual point, which is that the dependency should not exist at all.

Self-hosted queue stalls a required check: verify-precommit-config cancelled after ~50min queueing under PR burst #15407 — the stall, filed before I found this issue; should be closed into this one
ci(runners): move code-quality and the pre-commit gate onto the self-hosted runners (#15310) #15311 / CI runs entirely on GitHub-hosted runners while both self-hosted runners sit idle: 92 of 96 jobs target ubuntu-latest #15310 — the migration being superseded
ci(perf): scope enforce-precommit to the PR diff, not --all-files (#15358) #15392 — makes the worst offender cheaper, orthogonal to retirement

Owner decision, 2026-09-12: restrict the self-hosted runners, don't retire them. They become hardware-only and never gate. Every required check runs on GitHub-hosted runners, as this issue argues. Self-hosted runners take only jobs that need local hardware or very long suites, each with an explicit label, never publishing a required context and never running on a fork pull_request. This issue's evidence (nothing needs their hardware, and gating on a two-machine pool caused #13401, #15185 and #14444) is why no gate may use them. The policy, its CI guard and the SHA-pinning requirement are tracked in the new runner-policy issue.

Related, and deliberately not flagged as a duplicate — noting it from the v0.10.0 triage (#17863) so the relationship is recorded rather than rediscovered.

Retiring the self-hosted runners here would make #15309's runner-watchdog scheduling problem moot: there would be no self-hosted runner to watch. That makes these one decision with an ordering, not one defect — #15309 is a dependency of this issue's outcome, and if this lands first, #15309 may close without work rather than being done.

Verified in passing that this issue's own code criteria are met: runs-on: ubuntu-latest at code-quality.yml:97,132, enforce-precommit.yml:49,387 and five sites in frontend-test.yml, and git grep -E 'runs-on:.*self-hosted|- self-hosted' -- .github/workflows returns nothing.

The documented reason your last criterion asks for: four issues are blocked on the two-machine pool for non-CI verification

The CI argument here is sound and I am not contesting any of it — nothing in CI needs a self-hosted runner, and a self-hosted runner on a public repository is the documented anti-pattern. This is only about the final criterion:

Runners deregistered, or a documented reason they are kept for non-CI use

There is such a reason, and it is currently recorded on no issue. Four open issues are blocked on a multi-host topology, and three of them say so in their own words:

issue	its own wording
#17242	"Verified on a two-host* topology — a single-node run cannot reproduce this"*
#17243	"a single-node run cannot reproduce any of these"
#17331	"a single-node run cannot reproduce this, which is why…"
#10203	"a real two-machine verification run (CI can't cover distributed)"

#10203 is the sharpest of the four for this purpose, because it states the boundary this issue is drawing from the other side: CI cannot cover distributed verification. So "nothing in CI needs a self-hosted runner" and "nothing needs a two-machine pool" are different claims, and only the first is established above.

Why this is worth recording rather than discovering later

Three of those four carry blocked: needs-observation — "Nothing but an observation on a running system is left; no session can close it." Their gate is the rig, and the rig is Little-Slave + Second-Little-Slave. Deregistering before that is recorded converts four issues from blocked on an available affordance into blocked on an affordance that no longer exists, which is a worse state and an invisible one: nothing on those four issues names the pool, and nothing here named them.

That asymmetry is the general point. A gate can be written in either member of a pair, and this one was written in neither — the four issues say "a single-node run cannot reproduce this" without naming what could, and this issue counts CI contexts without counting non-CI dependents. Neither side is wrong; the edge simply was not recorded anywhere.

What I am proposing, and what I am not

Not contesting the retirement. Moving the 7 jobs to ubuntu-latest and retiring self-hosted-runner-health.yml are unaffected by this — those are about CI, which is settled.

Proposed for the last criterion specifically: take the second branch and record the reason, i.e. deregister from CI but keep the two machines available for distributed verification, with this comment as the documented reason and native blocked_by edges from #17242, #17243, #17331 and #10203 so the dependency is visible from either end. Then the first eight criteria proceed unchanged.

If the decision is to deregister entirely, that is a legitimate call and this comment is still the right place for it — but it should be made knowing it closes the only route four issues have, and those four should then be restated against something reachable rather than left reading as pending.

I have not established whether any other rig could serve them, so I am not claiming these two machines are the only option — only that they are the option that exists today and the one the four are implicitly waiting on.

Labelling needs-decision: the last criterion is the live question, and four issues depend on which branch is taken.

Runners deregistered, or a documented reason they are kept for non-CI use.

The second branch is not hypothetical and nobody has written the reason down, so the default path retires a capability four issues require. Recording this so the choice is made deliberately rather than by the first PR to implement the title.

The CI argument is not contested. Nothing in CI needs a self-hosted runner, and a self-hosted runner on a public repository is the documented anti-pattern. But "nothing in CI needs a self-hosted runner" and "nothing needs a two-machine pool" are different claims, and only the first is established here. This issue counts CI contexts; it does not count non-CI dependents. #10203 draws the boundary from the other side in its own words — "a real two-machine verification run (CI can't cover distributed)".

The four dependents, in their own words:

CRITICAL(deploy): backend provisioning aborts on every non-manager host — a task executes a script out of code_source, which exists only on the controller (rc=127) #17242 — "Verified on a two-host topology — a single-node run cannot reproduce this"
deploy: five more tasks read code_source on hosts that do not have it — three are in the builtin updater (#17242 class) #17243 — "a single-node run cannot reproduce any of these"
CRITICAL(deploy): the filtered backend manifest bakes a controller-only constraint path, so provisioning still aborts on every non-manager node #17331 — "a single-node run cannot reproduce this"
test(auth): cross-machine unified-identity + RBAC + two-host verification (#10193) #10203 — "a real two-machine verification run (CI can't cover distributed)"

Three of those four are CRITICAL. If the machines are deregistered and released, their acceptance criteria become permanently unsatisfiable — not delayed, unsatisfiable, because the thing that would satisfy them stops existing.

Recommendation: take the second branch — deregister from CI, keep the two machines for distributed verification. That delivers everything the CI argument asks for (no self-hosted runner attached to a public repo, no required context depending on the pool) while preserving the only environment in which four issues can be closed. The reason this issue invites is exactly the one above, and it costs nothing to record.

If full retirement is chosen instead, that is a legitimate call — but it should be made knowing it closes the verification route for those four, and they should be re-scoped or marked unverifiable in the same action rather than discovering it later.

Why this was not visible from either side. The four say "a single-node run cannot reproduce this" without naming what could; this issue counts CI contexts without counting non-CI dependents. Neither is wrong, and the dependency was written in neither member of the pair — so reading either one alone gives a complete-looking picture with the gate missing.

Correction: this decision was already made, on 2026-09-12. Removing `needs-decision`.

I labelled this needs-decision earlier today and recommended "deregister from CI, keep the two machines for distributed verification". That is already the recorded owner decision, and it has been for three weeks. I did not search for it before posting, which is the error.

#16463, under "Owner decisions, 2026-09-12":

Self-hosted runners are hardware-only and never gate. Every required check runs on GitHub-hosted runners… A self-hosted runner takes only jobs that need local hardware (NPU, GPU) or very long suites, and each such job carries an explicit label. It never publishes a required context and never runs on a pull_request event from a fork.

And its Why section cites this issue by number: "#15395 showed nothing in CI needs self-hosted hardware."

So the keep-door this issue holds open has already been walked through. The machines are retained for non-gating work; what is removed is their role as a CI gate. That is the second branch of this issue's last criterion — "a documented reason they are kept for non-CI use" — and #16463 is that document, with a stronger rationale than the one I offered: a self-hosted runner on a public repository that picks up untrusted pull-request code is a host compromise.

The dependent analysis stands and is unaffected. #17242, #17243, #17331 and #10203 each need a two-machine topology and each say so in their own words; three are CRITICAL. Under #16463's decision those machines continue to exist, so those four are not at risk — which is the opposite of what my earlier comment warned about, and the warning is withdrawn.

What remains here is implementation, not a decision: deregistering the runners from CI gating, per #16463's criteria — a guard failing any job that publishes a required context on self-hosted, and the policy written into the developer workflow docs.

Why I am recording the error rather than quietly editing the label. This is the fourth time today a session reached for "nothing owns this, so it needs a decision or a filing" without searching first, and the fourth time an owner already existed — after #17693 (defect class), #17890 (the braces exception), and this issue itself for the multi-host capability. My own stated rule from those three was search for the owner before filing, and I then applied it to filings and not to decisions. A needs-decision label on an answered question costs the same as a duplicate issue: it routes a human to a choice they already made.

Navigation Menu

Repository navigation

ci(runners): retire the self-hosted runners — nothing in CI needs them, and three required contexts depend on a two-machine pool #15395

What #15310 assumed

Evidence

Why hosted is strictly better here

Tasks

Acceptance criteria

Related

Discovered while moving the jobs: a second dead branch, not in the task list above

Live measurement: the self-hosted pool is the current queue bottleneck

Live evidence for this, from the session that caused it

What occurred

The detail worth keeping

Where I disagree slightly, and where I do not

Related

The documented reason your last criterion asks for: four issues are blocked on the two-machine pool for non-CI verification

Why this is worth recording rather than discovering later

What I am proposing, and what I am not

Correction: this decision was already made, on 2026-09-12. Removing `needs-decision`.

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

ci(runners): retire the self-hosted runners — nothing in CI needs them, and three required contexts depend on a two-machine pool #15395

Description

What #15310 assumed

Evidence

Why hosted is strictly better here

Tasks

Acceptance criteria

Related

Activity

mrveiss commented on Aug 31, 2026

Discovered while moving the jobs: a second dead branch, not in the task list above

mrveiss commented on Aug 31, 2026

Live measurement: the self-hosted pool is the current queue bottleneck

mrveiss commented on Sep 1, 2026

Live evidence for this, from the session that caused it

What occurred

The detail worth keeping

Where I disagree slightly, and where I do not

Related

mrveiss commented on Sep 12, 2026

mrveiss commented on Oct 2, 2026

mrveiss commented on Oct 3, 2026

The documented reason your last criterion asks for: four issues are blocked on the two-machine pool for non-CI verification

Why this is worth recording rather than discovering later

What I am proposing, and what I am not

mrveiss commented on Oct 3, 2026

mrveiss commented on Oct 3, 2026

Correction: this decision was already made, on 2026-09-12. Removing needs-decision.

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

Correction: this decision was already made, on 2026-09-12. Removing `needs-decision`.