ci: strict branch protection deadlocks when the self-hosted runner is offline — the branch-updater is pinned to that same runner #13401
Description
Dev_new_gui has branch protection with strict: true — "require branches to be up to date before merging". The only automation that brings a PR branch up to date is .github/workflows/auto-update-pr-branches.yml, and its auto-update job is pinned to the singleton self-hosted runner:
.github/workflows/auto-update-pr-branches.yml:29
runs-on: [self-hosted, linux]
So when that runner goes offline, the repository deadlocks:
- every open PR falls
BEHINDas soon as anything lands onDev_new_gui strict: truerefuses to merge aBEHINDbranch- the workflow that would clear
BEHINDcannot start, because it needs the offline runner - therefore nothing merges, and nothing ever will, until a human intervenes
Observed 2026-08-03: runner offline, 14 open PRs, 13 of them BEHIND, 10 with every required check already green and auto-merge armed. Two auto-update-pr-branches runs sat queued (15:00:57, 14:58:58) behind the outage. The last one to complete before the outage failed (13:29:12).
Why this is the wrong dependency
This is the same reasoning ci-dispatch-watchdog.yml already documents for itself, in a comment at the top of the file:
runs-on is ubuntu-latest on every job by design: the self-hosted runner is a singleton, and a watchdog that queues behind the outage it exists to report is useless.
A branch-updater that queues behind the outage it exists to recover from is useless in exactly the same way. The recovery mechanism must not share a failure domain with the thing it recovers.
Note the same file already gets this right for its other job — approve-refreshed-runs (:153) is ubuntu-latest. Only auto-update is pinned.
What the job actually needs
git fetch / git merge / git push against the repository, using GITHUB_TOKEN. There is no build, no test, no local toolchain, and no dependency on anything installed on the self-hosted host. There is no reason for it to be pinned.
Fix
Change :29 to runs-on: ubuntu-latest.
That alone breaks the deadlock: branches keep updating during a runner outage, PRs whose required checks are satisfiable on GitHub-hosted runners keep merging, and the outage degrades throughput instead of halting it.
Related, worth deciding at the same time
code-quality.yml:71 already carries runs-on: ubuntu-latest # Temporary: self-hosted runner offline (MVA-2605) — a previous outage forced the same move, and it was never reverted. frontend-test.yml has a unit-tests-skip shim (:195) publishing the required Unit & Integration Tests context from ubuntu-latest when no frontend paths changed. Both are the same lesson, learned twice, applied ad hoc.
The remaining self-hosted jobs are frontend-test.yml's unit-tests / security-scan / build-test / test-summary. Those genuinely run a build and a test suite, so pinning them is defensible — but unit-tests is a required context, which is what makes a runner outage block frontend PRs outright. Worth establishing whether it needs the self-hosted host at all: its steps are npm ci, npm run type-check, check-ts-delta and vitest. The full unit suite completes in ~164s on an ordinary machine (179 files, 2269 passed, 1 skipped, 0 failed, measured 2026-08-03).
Related: #11380 (singleton SPOF) · #13045 (outage leaves PRs pending) · #13388 (per-PR run count) · #12823 (parked runs)
Activity
fix(ci): cap the branch auto-updater at one PR per sweep and move it …
mrveiss commentedon Aug 3, 2026
Fixed by PR #13427 (batched with #13426 — same file, same job), squash-merged to
Dev_new_guiat 20:09:14.The
auto-updatejob is nowruns-on: ubuntu-latest(was[self-hosted, linux]), verified at line 46 on the base branch. The redundant "Install GitHub CLI" step (apt keyring +sudo) is gone, sinceghis preinstalled on GitHub-hosted runners.The deadlock this issue describes can no longer occur: the branch-updater no longer shares a failure domain with the runner whose outage is what makes branches go stale in the first place. Both jobs in that file are now GitHub-hosted, matching the principle
ci-dispatch-watchdog.ymlalready states for itself — "a watchdog that queues behind the outage it exists to report is useless."Closing manually with evidence —
Closes #Nonly fires on the default branch.ci(runners): move the last 7 jobs off self-hosted onto ubuntu-latest (#…