Skip to content

ci: strict branch protection deadlocks when the self-hosted runner is offline — the branch-updater is pinned to that same runner #13401

Closed

Description

@mrveiss

Dev_new_gui has branch protection with strict: true — "require branches to be up to date before merging". The only automation that brings a PR branch up to date is .github/workflows/auto-update-pr-branches.yml, and its auto-update job is pinned to the singleton self-hosted runner:

.github/workflows/auto-update-pr-branches.yml:29
    runs-on: [self-hosted, linux]

So when that runner goes offline, the repository deadlocks:

  1. every open PR falls BEHIND as soon as anything lands on Dev_new_gui
  2. strict: true refuses to merge a BEHIND branch
  3. the workflow that would clear BEHIND cannot start, because it needs the offline runner
  4. therefore nothing merges, and nothing ever will, until a human intervenes

Observed 2026-08-03: runner offline, 14 open PRs, 13 of them BEHIND, 10 with every required check already green and auto-merge armed. Two auto-update-pr-branches runs sat queued (15:00:57, 14:58:58) behind the outage. The last one to complete before the outage failed (13:29:12).

Why this is the wrong dependency

This is the same reasoning ci-dispatch-watchdog.yml already documents for itself, in a comment at the top of the file:

runs-on is ubuntu-latest on every job by design: the self-hosted runner is a singleton, and a watchdog that queues behind the outage it exists to report is useless.

A branch-updater that queues behind the outage it exists to recover from is useless in exactly the same way. The recovery mechanism must not share a failure domain with the thing it recovers.

Note the same file already gets this right for its other job — approve-refreshed-runs (:153) is ubuntu-latest. Only auto-update is pinned.

What the job actually needs

git fetch / git merge / git push against the repository, using GITHUB_TOKEN. There is no build, no test, no local toolchain, and no dependency on anything installed on the self-hosted host. There is no reason for it to be pinned.

Fix

Change :29 to runs-on: ubuntu-latest.

That alone breaks the deadlock: branches keep updating during a runner outage, PRs whose required checks are satisfiable on GitHub-hosted runners keep merging, and the outage degrades throughput instead of halting it.

Related, worth deciding at the same time

code-quality.yml:71 already carries runs-on: ubuntu-latest # Temporary: self-hosted runner offline (MVA-2605) — a previous outage forced the same move, and it was never reverted. frontend-test.yml has a unit-tests-skip shim (:195) publishing the required Unit & Integration Tests context from ubuntu-latest when no frontend paths changed. Both are the same lesson, learned twice, applied ad hoc.

The remaining self-hosted jobs are frontend-test.yml's unit-tests / security-scan / build-test / test-summary. Those genuinely run a build and a test suite, so pinning them is defensible — but unit-tests is a required context, which is what makes a runner outage block frontend PRs outright. Worth establishing whether it needs the self-hosted host at all: its steps are npm ci, npm run type-check, check-ts-delta and vitest. The full unit suite completes in ~164s on an ordinary machine (179 files, 2269 passed, 1 skipped, 0 failed, measured 2026-08-03).

Related: #11380 (singleton SPOF) · #13045 (outage leaves PRs pending) · #13388 (per-PR run count) · #12823 (parked runs)

Activity

  1. mrveiss commented on Aug 3, 2026

    @mrveiss
    OwnerAuthor

    Fixed by PR #13427 (batched with #13426 — same file, same job), squash-merged to Dev_new_gui at 20:09:14.

    The auto-update job is now runs-on: ubuntu-latest (was [self-hosted, linux]), verified at line 46 on the base branch. The redundant "Install GitHub CLI" step (apt keyring + sudo) is gone, since gh is preinstalled on GitHub-hosted runners.

    The deadlock this issue describes can no longer occur: the branch-updater no longer shares a failure domain with the runner whose outage is what makes branches go stale in the first place. Both jobs in that file are now GitHub-hosted, matching the principle ci-dispatch-watchdog.yml already states for itself — "a watchdog that queues behind the outage it exists to report is useless."

    Closing manually with evidence — Closes #N only fires on the default branch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions