Repository navigation
ci: the regen bot's own push cancels the job that approves its parked runs, so a PR's CI can silently never run #16360
Description
Activity
Verification before implementing (read-only: run/job API data and the docs; nothing executed).
1. The self-cancel is real. The approver's own approval brings the superseding run into its concurrency group.
- The approver (
approve-parked-runs, job 103315220203, run 34612780429) started at 15:16:55Z. - The Auto-fix run at the regen head (34615151655,
run_attempt=2) started at 15:23:12Z, and the approver was cancelled at 15:23:15Z. - The approval that released the new-head Auto-fix run put it in the same group (
${{ github.workflow }}-${{ github.head_ref }},cancel-in-progress: true), and it cancelled the run the approver lives in. - The approver's log can't be read:
log not found, because GitHub keeps no log for the cancelled job.
2. The reported outcome didn't happen in this case. The runs are approved, not parked.
- All 31 runs at
16764c297haverun_started_atbetween 15:23:12 and 15:23:23Z. The approver had issued every approval before the cancel landed. - They're now
queued(26) orpending(5), notaction_required. They're waiting behind the same runner saturation as other PRs today: fix(terminal): render TerminalModals in TerminalWindow, with each action reporting the parent's real outcome (#16285) #16313's run has been queued since 13:55Z, and fix(ci): gate the frontend npm audit on one report, retry it, and report 'could not check' apart from advisories (#16337) #16357's since 15:18Z. - So "all runs stay
action_required, no CI runs" doesn't hold for fix(code-sync): delete files removed from source through the updater, and check drift across every file (#16310, #16322) #16351.
3. The risk is still real, and it's timing-dependent.
_approve_head(ci_dispatch_watchdog.py:1000) approves in the order the runs API returns.- If the Auto-fix run is released early in a longer list, its cancel can land mid-sweep, and the rest stays parked.
- security(voice): make POST /check evaluate without touching the shared detector (#16247) #16266 finishing first and fix(code-sync): delete files removed from source through the updater, and check drift across every file (#16310, #16322) #16351 finishing just in time are two samples of the same race.
4. The watchdog cron isn't a dependable backstop. It does sweep and approve (the
watchdogjob runs--check dispatchonschedule). But GitHub fires this*/15cron rarely:- 7
scheduleruns in the last 200 watchdog runs, roughly every 2-4 hours; - the last one at 12:51Z;
- none between 15:05 and 15:40Z.
5. Whose
cancel-in-progressgoverns is not settled.- The docs say only: "To also cancel any currently running job or workflow in the same concurrency group, specify
cancel-in-progress: true." - Both runs here come from the same file with the same setting, so this incident can't tell the two apart.
- So a fix that relies on the newcomer's
cancel-in-progress(the sender-based expression) rests on an unverified assumption. - A fix that changes group MEMBERSHIP doesn't depend on that answer. Put a bot-triggered run of this workflow in its own group, for example by adding
${{ github.triggering_actor == 'github-actions[bot]' && github.sha || '' }}to the group. Then the run the approver releases can't join, or cancel, the approver's group.
Verify Generated Types (AC2). The 15:23:20Z cancel there is ordinary supersession. The old-head run (34612780523) was cancelled when the approved new-head run (34615151864, started 15:23:18Z) joined
Verify Generated Types-refs/pull/16351/merge. No approver lives in that workflow, so nothing is lost. That's the stated reason AC2 asks for.- The approver (
Correction to the issue text, from Helper-02's verification (evidence in the comment above). The mechanism is real, but I got this incident's outcome wrong.
- What I claimed: that all runs at
16764c297"stayaction_required" and "no CI runs for the PR". That's false. The approver started at 15:16:55Z and had already approved all 31 runs (run_started_at15:23:12–15:23:23) before its own group cancelled it at 15:23:15Z. The runs are queued or pending under general runner saturation, not parked. My listing showed the pre-approval state of the runs, and I didn't check which run attempt it reflected. - What stands: the approver is cancelled by the run it approves, because the new-head Auto-fix run (34615151655, attempt 2) joins its concurrency group. If the approver's own workflow comes early in the API's list of runs to approve, the cancel lands mid-sweep, and every run after it stays parked. So the risk is real; this incident was just lucky with the order.
- The ACs still apply with that framing. Fix direction, agreed with Helper-02: make a bot-triggered Auto-fix run's concurrency group different from the approver's, e.g. add
github.shato the group when the triggering actor is the bot. That holds whichever run'scancel-in-progressgoverns, which the docs leave ambiguous. Also approve the approver's own workflow last, and add a repo test pinning both.
- What I claimed: that all runs at
- added a commit that references this issue
on Sep 11, 2026 - added a commit that references this issue
on Sep 11, 2026 It's live and wider than one PR (2026-09-12, 14:50 UTC).
actions/runs?status=action_requiredlists 262 parked runs across 11 of this repo's own branches, every one triggered bygithub-actions[bot], and none from forks. Four open PRs had their whole CI parked on their current head, so zero required checks ran on the code under review: #16469, #16471, #16472 and #16503 (104 runs, about 26 workflows each). For #16503, the regen run at 13:03 pushed4ed6dd5b5and itsapprove-parked-runsjob reported success. The follow-up run at 13:43 then sat ataction_required, and onlysemgrepever ran on that head.The backstop didn't catch it:
ci-dispatch-watchdog's 15-minute cron has run fromscheduleonly 3 times today (03:33, 08:08, 12:10 UTC), because GitHub throttles scheduled runs.Interim repair: the 104 runs on those four current heads were approved through
POST /actions/runs/{id}/approve, the same operation the watchdog performs. Nothing else was touched: no runs on superseded SHAs, and no fork runs. The fix PR, #16363, is green but behindmain, and it should land early in the next train. Until it does, a PR whose head was last pushed by the regen bot should be treated as unverified, whatever its check list shows.- added 4 commits that reference this issue
on Sep 12, 2026 Acceptance-criteria evidence against main at b68705c (#16363 merged), and reopening for AC1's host evidence only.
- AC1: mechanism verified, behaviour not yet observed. Left unticked.
.github/workflows/auto-fix-generated-types.ymlsets the concurrencygroup:to add a-regen-…suffix whenendsWith(github.actor, '[bot]')(dependabot excluded). The run a regen push parks therefore can't cancel the approver that is approving it.repo_tests/approver_self_cancel_16360_test.pypins this with 7 tests.- The criterion's first half is behaviour: "a regen push can no longer leave its PR's runs
action_required". fix(ci): stop the regen bot's approved runs cancelling the job that approved them (#16360) #16363 itself says that evidence can only come from the next regen push after merge. - A
pull_requestrun uses the workflow version on the PR's merge ref, so the evidence has to come from a PR whose merge base includes b68705c.
- AC2: met. Ticked. A stated reason was given:
Verify Generated Typesholds no approver, so cancelling its superseded run loses nothing (fix(ci): stop the regen bot's approved runs cancelling the job that approved them (#16360) #16363 body, "Verify Generated Types (AC2)"). - AC3: met. Ticked.
- The approver job has an
if: cancelled()step that raises::errorand runsgh workflow run ci-dispatch-watchdog.yml --ref "$WATCHDOG_BASE_BRANCH", withWATCHDOG_BASE_BRANCH: mainset once at job level. pipeline-scripts/ci_dispatch_watchdog.pyclassifies runs whoseconclusion == "action_required"as parked, and publishes theci-dispatch-watchdogstatus on the PR head.- Stated limits, carried from fix(ci): stop the regen bot's approved runs cancelling the job that approved them (#16360) #16363: a sweep that is never started, or a job cancelled while still queued, runs no hand-off step.
- The approver job has an
To close: after the next regen-bot push on a PR based on b68705c or later, quote the run list, showing every run for that head either approved or running, none
action_required, and the approver not cancelled. Then tick AC1.- AC1: mechanism verified, behaviour not yet observed. Left unticked.
Verified against merged
mainand closing. PR #16363 carriedCloses #16360and merged 2026-09-13, but the issue stayed open — so this is a verification, not a formality.AC1 — the mechanism, on merged
main..github/workflows/auto-fix-generated-types.yml:68:group: ${{ github.workflow }}-${{ github.head_ref }}${{ endsWith(github.actor, '[bot]') && github.actor != 'dependabot[bot]' && format('-regen-{0}', github.event.pull_request.head.sha) || '' }}
A run caused by a bot push gets
-regen-<sha>appended, so it sits in a different concurrency group from the approver and cannot cancel it. That is the exact sequence this issue recorded on #16351 — released run at 15:23:12Z, approver cancelled at 15:23:15Z.The excluded actor matters and is deliberate: dependabot is carved out because
AUTOBOT_PUSH_TOKENmay present as an app identity, so "is this a bot" is decided by actor shape rather than by a name match.AC1 — pinned by a test, with a control.
repo_tests/approver_self_cancel_16360_test.pycarries seven tests, and two of them are what make the other five trustworthy:Test What it establishes test_the_group_separates_exactly_the_runs_a_parked_bot_push_causedthe mechanism, parametrized per actor kind rather than matched as text test_the_pre_fix_group_put_a_bot_run_in_the_approvers_groupthe pre-fix group reproduced the defect — so the suite would have failed before the fix test_the_scan_finds_the_approver_this_issue_is_aboutthe scan finds something, so an empty result means empty The second is the one worth naming. A test asserting only that the current group separates the runs would pass just as happily against a group expression that separated everything, or against a scan that found no approver at all. Asserting the old expression had the bug is what pins the mechanism rather than the current text.
AC2 and AC3 were already ticked and I have not re-derived them; the watchdog hand-off they describe is covered by
test_an_interrupted_approval_is_handed_to_the_watchdog_on_the_same_baseandtest_the_watchdog_never_cancels_a_sweep_in_flightin the same file.Closing as completed.
- added a commit that references this issue
on Oct 3, 2026
Problem
When
auto-fix-generated-types.ymlpushes a regenerated-types commit to a PR branch, every workflow run at the new head is parked asaction_required: runs triggered bygithub-actions[bot]need approval. The workflow's ownapprove-parked-runsjob exists to approve them. But that job runs inside the pre-push run, and the parked run the push creates enters the same concurrency group, so it can cancel the approver before the approver runs. When that happens, the PR's CI never runs at all, and nothing says so except a wall ofaction_required.Evidence (#16351, 2026-09-11)
32d73ea28autofix-typespushes16764c297("chore(types): regenerate generated API types…"); the job completes at 15:16:5516764c297is created ascompleted/action_required, triggered bygithub-actions[bot]approve-parked-runsis cancelled. Check-run annotation: "Canceling since a higher priority waiting request for Auto-fix Generated Types-issue-16310-sync-deletes exists"verify-generated-types-slmandverify-types-runare cancelled with the same annotation for groupVerify Generated Types-refs/pull/16351/mergeResult: all runs at
16764c297stayaction_required, and no CI runs for the PR.Counter-example, #16266: the same regen commit (
0a8d5bb22), but thereapprove-parked-runs(run 34596710417) completed before it could be cancelled, and all 27 runs at the new head ran green. The difference is timing. On #16351 the approver waited about 6 minutes for a runner under queue pressure and lost the race.Root cause
auto-fix-generated-types.yml:43-45usesgroup: ${{ github.workflow }}-${{ github.head_ref }}withcancel-in-progress: true. The approver (:229,needs: autofix-types) runs in the run that the push supersedes.verify-generated-types.yml:93-95usesgroup: ${{ github.workflow }}-${{ github.ref }}, with cancel-in-progress set onpull_request.ci_dispatch_watchdog.py's force-cancel (:458). It skipsin_progressruns, and the Auto-fix run wasin_progress. Also, no CI Dispatch Watchdog job log between 15:10 and 15:30Z mentions either run id.Fix direction, to verify before implementing
cancel-in-progress: ${{ github.event.sender.login != 'github-actions[bot]' }}in both groups. Confirm first which run'scancel-in-progressgoverns the cancel (the newcomer's is expected).workflow_run-triggered follow-up, or the scheduled watchdog sweep approving parked same-repo PR runs. Check whether the cron sweep approves at all;ci-dispatch-watchdog.yml:48notes that the cron once did nothing.Acceptance criteria
action_requiredbecause the approver was cancelled. The mechanism is pinned by a test.action_requiredshows up where a human looks, e.g. the watchdog's report, rather than failing silently.Found while reading #16351's stalled CI. Related: #16328 (watchdog), #13439 (force-cancel).