Skip to content

fix(ci): stop one main commit's CI from cancelling another's - #281

Merged
emrecdr merged 1 commit into
mainfrom
fix/ci-main-runs-cancel-each-other
Aug 16, 2026
Merged

fix(ci): stop one main commit's CI from cancelling another's#281
emrecdr merged 1 commit into
mainfrom
fix/ci-main-runs-cancel-each-other

Conversation

@emrecdr

@emrecdr emrecdr commented Aug 16, 2026

Copy link
Copy Markdown
Owner

Root-cause fix for the cancelled CI runs observed on 6382f11 and 0eedd37 tonight.

What happened

Three PRs merged within ~23 minutes (#278, then #279 and #280 auto-merging). Each left the previous commit's CI cancelled, so two commits on main are permanently unverified.

The cause is not the merges. merge-on-green must re-dispatch main's CI after every auto-merge, because a GITHUB_TOKEN push does not fire on: push — without the dispatch an auto-merged commit gets no CI at all. That dispatch is correct. The defect is that it joined concurrency group CI-refs/heads/main, where cancel-in-progress: true made it kill the previous commit's still-running matrix.

The old comment reasoned the risk away:

"For main, sequential commits would also cancel each other, but main commits rarely arrive that fast and the more recent commit is what callers care about anyway."

Both halves are wrong. They arrive that fast by construction, and on a protected branch the older commit is not disposable — self-gate stamps it and dogfood publishes from it.

The fix, and why the obvious one is insufficient

The ecosystem-standard fix is cancel-in-progress: ${{ github.ref != 'refs/heads/main' }}. It would not have fixed this. GitHub keeps at most one running and one pending run per group and cancels the older pending one — so with three rapid merges the middle commit still loses its run. That is exactly tonight's case, so that fix would have masked the defect rather than removed it.

Qualifying the group by github.sha on main is the complete fix — each commit gets its own group and never contends:

Case Group Shares?
main commit A vs B CI-refs/heads/main-<shaA> / -<shaB> No — never cancel each other
PR push 1 vs 2 CI-refs/pull/N/merge- (both) Yes — cancellation preserved
tag dispatch CI-refs/tags/vX- own group

cancel-in-progress stays true because it remains correct within a group: on a PR it supersedes the stale push, and on main it only ever sees a deliberate re-dispatch of the same SHA.

Scope audit

Checked all seven workflows. Only ci.yml had the defect — container.yml/release.yml are tag-triggered so their groups are already unique per release, bench.yml already sets cancel-in-progress: false, and the two reusable workflows declare no concurrency.

Guard

workflow_concurrency_test.rs pins the property and proves itself rather than only passing: it rejects the exact ref-only group that caused this, accepts the shipped one, and asserts its pinned copy still equals what the workflow ships so the two cannot drift apart.

Verification

  • cargo fmt --all --check — clean
  • Guard test — 3/3 pass against the shipped config
  • YAML parsed and the group expression simulated across main/PR/tag refs

The full local gate could not be re-run: the earlier ENOSPC left the shared cargo target directory with dangling fingerprints (extern location for gix_trace does not exist, then regex-automata). Clearing it is a shared-cache decision I have not taken unilaterally, so CI carries that verification.

The concurrency group was scoped to the ref, so every push and dispatch
on main shared `CI-refs/heads/main` and each new one cancelled the last.

On a pull request that is correct -- a new push makes the previous run
worthless, and cancelling saves runner minutes. On main it is not.
Every commit there is a permanent, releasable point that `self-gate`
stamps and `dogfood` publishes from, and a commit whose run was
cancelled is unverified forever because no later run covers it.

The workflow comment reasoned the risk away on the grounds that "main
commits rarely arrive that fast". They do, by construction:
`merge-on-green` must re-dispatch main's CI after every auto-merge,
because a GITHUB_TOKEN push does not fire `on: push` and the merge
commit would otherwise land with no CI at all. Three merges landing
together therefore had each dispatch kill the previous commit's
still-running matrix, and two commits went unverified.

Qualifying the group by `github.sha` on main gives each commit its own
group, so nothing contends. `cancel-in-progress: false` would NOT have
been enough: GitHub keeps at most one running and one *pending* run per
group and cancels the older pending one, so the middle commit of three
would still have lost its run. Distinct groups are the only thing that
works, which is also why the guard checks the group rather than the
cancel flag.

Behaviour is unchanged everywhere else. Pull-request pushes still share
a group and still supersede their predecessors; a deliberate
re-dispatch of the same main SHA still replaces the stale run. Audited
the other six workflows: container and release are tag-triggered so
their groups are already unique per release, bench already sets
`cancel-in-progress: false`, and the two reusable workflows declare no
concurrency.

The guard proves itself rather than only passing: it rejects the exact
ref-only group that caused this, accepts the shipped one, and asserts
its pinned copy still equals what the workflow ships so the two cannot
drift apart.
@emrecdr
emrecdr merged commit 1b93827 into main Aug 16, 2026
14 checks passed
@emrecdr
emrecdr deleted the fix/ci-main-runs-cancel-each-other branch August 16, 2026 20:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants