fix(ci): unhang the required frontend check, and make an unhung one detectable (#13341, #13045, #13286)

[mrveiss]

Thinking Path

Three issues, one shape: CI can fail in a way that produces no signal, and every
guard added so far has itself been unobserved.

• ci: required frontend check hung 3h past its 30-minute timeout, blocking every merge #13341 is a job that hung; the timeout that existed to stop it is enforced by the
  runner that stopped responding.
• ci: runner outage leaves PRs indefinitely pending with a required context that never reports — reads as '19 success, 0 failures' #13045 is a required context that never reports; its replacement probe reads a hung
  job as proof of health, so ci: required frontend check hung 3h past its 30-minute timeout, blocking every merge #13341 would have made it report green.
• Marker-excluded tests (slow/integration/performance/distributed) run in no workflow at all #13286 is a nightly suite that has never executed once.

So the three are not adjacent, they are the same defect at three layers: fix the hang,
make the hang detectable, and make the detectors actually run.

What Changed

#13341 — root cause of the hang

Check TypeScript error delta ran over three hours against timeout-minutes: 30.
Its sibling Run type checking — the same vue-tsc, same job, reached through
npm run — finished in about a minute.

The difference was npx. check-ts-delta.sh invoked npx vue-tsc; type-check
invokes the binary npm already put on PATH. Everything npx adds before running a
binary is an unbounded wait:

• it resolves the package first, which can reach the registry;
• when resolution misses it prompts (Ok to proceed?) and blocks on stdin, and a
  CI step's stdin is a pipe that never delivers a line and never closes;
• it takes locks under the npm cache, which on the self-hosted runner is shared with
  every concurrently executing job instead of being a fresh VM per job.

vue-tsc is a devDependency and is on disk after npm ci, so there was nothing for a
resolver to do. The script now calls the installed binary directly with </dev/null,
removing the network path, the prompt and the cache lock together. That accounts for
the intermittency (a lock or a fetch, not a code path) and for why only this one of
the two vue-tsc steps was affected.

A fail-open surfaced while fixing it. || true swallowed the compiler's exit
code, so a vue-tsc that crashed without emitting diagnostics counted zero errors
and reported PASS. That is the documented OOM mode of this exact step, so the gate was
open in precisely its known failure case. Non-zero exit with zero diagnostics is now a
hard failure.

The step is additionally wrapped in timeout --kill-after=60s 600. GitHub enforces
timeout-minutes from the runner side, so a runner that stops reporting progress may
never receive the cancellation; timeout runs inside the step's own process tree and
fires regardless. This bounds the next unknown cause — it is not the fix for this
one.

#13341 + #13045 — making a wedged job visible

The starvation probe on base was not merely blind to #13341, it was inverted by
it. A wedged job is status=in_progress on a self-hosted label, which is exactly
what self_hosted_pool_is_serving read as proof the pool was alive. Work queued
behind the hung job was reported as "contention, not an outage". The reviewer's
false-negative concern was correct, and this is the concrete mechanism.

inspect_self_hosted_pool now separates liveness from wedging in the same job
listing, at no extra API cost. A self-hosted job still executing after 45 minutes is
excluded from the liveness verdict and returned as an OverdueJob. 45 is derived, not
chosen: every self-hosted job in .github/workflows declares timeout-minutes of 30
or less, so nothing merely slow can reach it while still inside its own limit.

Published where it is actionable:

• --check dispatch (every PR event, every base push) writes a failing commit
  status naming the wedged job on the head it is blocking — the fault appears on the
  pull request, not only in the Actions tab.
• --check runner-starvation exits non-zero on a wedged job even with an empty
  queue. A job holding a required context is a fault when it happens, not once a
  queue has built behind it. Waiting for the queue would reproduce the three-hour
  silence.

Two limits are documented rather than papered over: queued runs cannot be
attributed to a runner pool from the API (labels live on jobs, and a starved run has
no jobs — that absence is the condition being detected), and the half-hourly
schedule still executes the pre-#13045 copy of this workflow that lives on the default
branch. A push trigger on the integration branch gives the repaired probe live
coverage meanwhile.

#13286 — the nightly that never ran

Not "ran and failed" — zero runs of any kind:

$ gh run list --workflow=marker-tests.yml
failed to get runs: could not find any workflows named marker-tests.yml

GitHub registers schedule and workflow_dispatch from the default branch only.
The file exists on the integration branch and not on the default branch, which is 72
commits behind at time of writing (it drifts as PRs merge; the integration branch is
in turn 1 commit behind the default branch), so neither declared trigger can dispatch. A nightly that cannot be
scheduled runs exactly as many marker-excluded tests as no workflow at all.

Added push + pull_request triggers scoped to the workflow's own file, so it is
exercised on the PR that changes it and can never again ship unexecuted. The cron is
kept — it is correct, and starts working on promotion.

Runner move: recommended, deliberately NOT done here

Unit & Integration Tests is a required status check, and moving it changes which
runner that requirement depends on — so it is not a silent change.

Recommendation: move unit-tests, security-scan and build-test to
ubuntu-latest. They are npm/vue-tsc/vitest steps with no self-hosted
dependency; the shim job proving the required context already runs on hosted runners,
so the required context is demonstrably satisfiable there. It removes the entire class
of "one hung frontend job blocks all backend merges", it removes the shared-$HOME
npm-cache contention that is the most probable trigger of this very hang, and it is
the reasoning that moved the Python gate to hosted runners.

Why not in this PR: the file's opening line states self-hosted was chosen to
avoid GitHub Actions quota limits. Spending billable minutes on the largest job in
the repository is a cost decision for the owner, not a reviewer-level one, and it is
separable from the correctness fixes here. Left as an owner decision with the
reasoning recorded.

Review amendments

Both review findings were cases where an earlier commit's stated intent was not
actually implemented.

F1 — the fail-open fix gated on the diagnostic count, which is precisely the number
that is unreliable when the compiler does not complete. Two failures still passed,
reproduced against the real vue-tsc: a missing tsconfig.app.json emits ONE
diagnostic (TS5058) having checked nothing, giving PASS: 1 errors (187 below baseline); and a crash after partial diagnostics gave PASS: 5 errors. The gate is
now the exit status — vue-tsc returns TypeScript's ExitStatus, measured 2 for a
real type error, 1 for a configuration diagnostic, 3/4 for an invalid project — so
> 2, or non-zero with no diagnostics, plus an explicit TS5xxx/TS6xxx check for
configuration diagnostics reported instead of a type check.

F2 — the wedged-job detector could not have caught the incident it was written for.
It truncated to the 10 newest in-progress runs, but GET /actions/runs returns
newest-first and a wedged run is by definition the oldest. The full page is now
listed (one call either way) and sorted oldest-first before the per-run job-lookup
budget is spent; the budget is raised from 10 to 25, above the observed steady state.
The unit tests passed throughout because a synthetic listing has nothing to truncate —
the new test fails against the pre-fix implementation.

frontend-typecheck-regression.yml carried the identical fail-open, untouched and
untested: set +e plus a bare grep -c meant a crashing compiler produced ERRORS=0,
which against BASELINE: 0 is a pass. Same fix shape applied, and npx dropped there
too. (Its config-error case happened to fail for the wrong reason — TS5058 counts as
one error against a baseline of zero — so the fail-open there is specifically the crash
path, which is this project's documented OOM mode.)

Also timeout --verbose, so a bound step prints timeout: sending signal TERM to command 'npm' rather than leaving the operator to decode a bare exit code 124.

Verification

Ran locally:

Check	Result
actionlint (whole repo, shellcheck enabled)	exit 0, no output
shellcheck autobot-frontend/scripts/check-ts-delta.sh	clean
bash -n on the script	OK
pytest repo_tests/ci_dispatch_watchdog_test.py	91 passed (75 before; 16 added)
black --line-length=120 --check	2 files unchanged
flake8 --config=.flake8	0

check-ts-delta.sh behaviour, against a stubbed compiler (all five observed):

Scenario	Result
10 errors vs baseline 42	PASS, exit 0
50 errors vs baseline 42	FAIL, exit 1
crash, exit 134, no diagnostics	exit 1 — pre-change script gave PASS: 0 errors, exit 0
compiler reads stdin	returns immediately, does not hang
binary absent	explicit error, exit 1 (not a silent pass)

timeout --kill-after=60s verified to terminate the whole child process group
(exit 124, no leaked processes).

Amendment evidence — check-ts-delta.sh against the REAL vue-tsc 5.9.3:

Scenario	raw vue-tsc	before	after
clean sources	exit 0	pass	pass
one genuine type error	exit 2	pass (1 < baseline)	pass (1 < baseline)
tsconfig.app.json missing	exit 1, TS5058	PASS: 1 errors, exit 0	exit 1
5 diagnostics then crash	exit 134	PASS: 5 errors, exit 0	exit 1
crash, no diagnostics	exit 134	PASS: 0 errors, exit 0	exit 1
invalid project	exit 3	pass	exit 1
300 errors vs baseline 188	exit 2	exit 1	exit 1
compiler reads stdin	—	hangs	returns immediately

frontend-typecheck-regression.yml, step extracted from the YAML and executed
against the same real vue-tsc: clean → exit 0; one real type error vs BASELINE: 0 →
exit 1; crashing compiler → base gives ✅ No regression: 0 == 0, exit 0, fixed gives
exit 1.

F2 ordering, test_the_wedged_run_is_found_even_when_it_is_the_oldest_of_many:
fails against the pre-fix implementation (Right contains one more item: 'wedged'),
passes after. 92 tests total (75 on base, 17 added).

Fork-PR guard byte-unchanged — AST-sliced SHA-256 against base:

is_approvable      e638f88fcbd2757a   (base: e638f88fcbd2757a)
run_is_same_repo   c1c87b2492c91e70   (base: c1c87b2492c91e70)
is_parked          f4bb0c79a6b08110   (base: f4bb0c79a6b08110)

New tests pin the behaviour that matters, including the regression itself —
test_a_queue_behind_a_wedged_job_is_no_longer_dismissed_as_contention fails against
the base implementation.

Not verifiable without a live run — stated plainly:

• That the frontend step no longer hangs. The hang is intermittent (a re-run of the
  same commit passed the step cleanly), so a single green run proves nothing; only an
  absence over many runs will. The timeout bound means the next occurrence fails at
  a known point with a diagnosable message instead of hanging, which is the property
  actually being delivered.
• The 3h05m job's own records are no longer retrievable (that attempt has been
  superseded), so the measurement is taken from the issue rather than re-derived.
• Whether marker-tests.yml goes green. This PR touches it, so the new
  pull_request trigger fires it here — that run is the point, and its outcome is
  the first observation this workflow has ever produced.
• Whether a real wedged job produces the commit status. The logic is unit-tested
  against synthetic API responses; the write path needs a live occurrence.
• The half-hourly schedule keeps running the default branch's copy of the health
  workflow until promotion, so its red runs will continue regardless of this PR.

Model Used

Claude Opus 4.5 (claude-opus-4-5-20251101)

---

Closes #13341
Refs #13045 — the probe is repaired and regression-tested, but the issue's user-visible
symptom is a health check that never reports, and after this merges the */30
schedule still runs the default branch's broken copy and still fails every time. The
repaired probe gets push-time coverage only.
Refs #13286 — the workflow now has a trigger that fires and is exercised on this PR,
but the nightly cadence cannot dispatch until the file reaches the default branch.

[github-actions]

✅ SSOT Configuration Compliance: Passing

🎉 No hardcoded values detected that have SSOT config equivalents!

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!