ci: cancel-in-progress voids the base-branch suite — no post-merge verification on Dev_new_gui

The base branch is essentially never verified

ci.yml uses:

concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

For a pull request that is correct — a new push should cancel the superseded run. For a push to Dev_new_gui it means each merge cancels the previous merge's verification. Because merges arrive faster than the suite finishes, most merges are never verified at all.

Measured

23 of the last 38 completed ci.yml runs on Dev_new_gui were cancelled (61%). The other 15 were failure. Not one was success.
17 merges to Dev_new_gui in three hours — one every ~10.6 minutes.
The sharded suite takes ~9 minutes.

A ~10.6-minute merge interval against a ~9-minute run means cancellation is the normal outcome, not the exception. At one point eight consecutive runs were cancelled.

Why this blocks #13162

#13162 asks for python-suite to become a required status check. That cannot safely happen while this holds: a required check that the next merge cancels would leave PRs permanently un-mergeable, blocking the repo rather than protecting it.

Sharding (#10691) fixed the duration problem #13162 opened with — the suite went from ~51 minutes to ~9. It did not touch the cancellation problem, and cancellation is now the binding constraint. This should land before the required-check change, not after.

It also has a quieter cost today: nobody can tell whether Dev_new_gui is green, because the measurement is destroyed before it completes. Re-measuring the #13162 backlog has repeatedly required hunting for a run that happened to survive.

Proposed fix

concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: ${{ github.event_name == 'pull_request' }}

PR runs keep cancelling on supersede; base-branch runs are allowed to finish.

The trade-off, stated honestly

This is not free, which is why it is filed rather than just applied:

Base runs would queue instead of cancel. At ~17 merges per 3 hours that is ~17 additional runs to completion in that window.
The python-suite shards run on ubuntu-latest, so they cost the self-hosted pool nothing. But ci.yml also has jobs on the [self-hosted, Linux, X64] singleton, and those would accumulate. The runner was offline earlier today and a ~60-run backlog built up, so headroom there is not generous.

Options if the straight fix is too costly:

Separate concurrency group for the base branch — let base runs serialize among themselves without cancelling, while PR runs keep the current behaviour.
Restrict the change to the python-shard job only (job-level concurrency), so the expensive self-hosted jobs keep cancelling and only the ubuntu-latest suite is preserved. This is the narrowest change that makes the suite measurable.

Option 2 is probably the best value: it makes python-suite trustworthy as a gate without adding any load to the singleton pool.

Acceptance criteria

A merge to Dev_new_gui no longer cancels the previous merge's python-suite run
PR runs still cancel on supersede
Self-hosted queue depth does not materially worsen (measure before/after)
A comment in ci.yml records why base and PR differ, so cancel-in-progress: true is not restored wholesale later

Provenance

Found while measuring the remaining #13162 backlog: eight consecutive base runs were cancelled, making the measurement impossible without hunting for a surviving run. Related: #13162, #10691.

Correction: this does not block the #13162 required-check change

I claimed in the description that making python-suite required is unsafe while this holds. That was wrong, and the reasoning was wrong in a specific way worth recording.

Required status checks are evaluated on the pull request head, not on the base branch. The concurrency group is ${{ github.workflow }}-${{ github.ref }}, and a PR's ref is its own — so another session merging to Dev_new_gui does not touch a PR's run. Checked against the last 12 pull_request runs: they complete success/failure. The only cancellations are repeated pushes to the same PR (issue-13398 shows three, all supersedes), which is exactly what cancel-in-progress is for.

So python-suite could become a required check today without this being fixed. The remaining blocker for #13162 is simply that the suite is not yet green.

What this issue actually costs — still worth fixing

The problem is real, just narrower than I described:

Nobody can tell whether Dev_new_gui is green. 23 of the last 38 completed base runs were cancelled and none succeeded.
Post-merge regressions go unobserved. A defect that only appears once two PRs are combined has no run that would catch it.
Measuring the backlog means hunting for a run that happened to survive — that cost real time during the ci: Python unit-test suite is not a required check — red tests merged to Dev_new_gui twice #13162 campaign.

That is a monitoring and post-merge-verification gap, not a merge-gating one.

A correction to the proposed fix, too

I suggested job-level concurrency on python-shard as the narrowest option. That does not work. Workflow-level cancel-in-progress cancels the entire run including its jobs; a job-level concurrency group cannot override that. The options are actually:

cancel-in-progress: ${{ github.event_name == 'pull_request' }} — base runs stop cancelling, but with a shared group they queue behind each other.
Give push events a per-SHA group so base runs neither cancel nor queue:
```
group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event_name == 'push' && github.sha || '' }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
```
Base runs then execute concurrently. The cost lands on the self-hosted singleton jobs in ci.yml, which would accumulate.
Move python-shard into its own workflow file with its own concurrency policy — the only option that isolates the ubuntu-latest suite from the self-hosted jobs entirely.

Option 3 is the one that actually achieves "the base suite always produces a verdict" without adding self-hosted load. It is a larger change than a one-line edit.

Separate throughput note for #13162

Branch protection has strict: true. If python-suite becomes required, every merge to Dev_new_gui forces every open PR to update and re-run the full suite. At ~17 merges per three hours that is substantial additional CI. Worth sizing before flipping the switch — it is a separate concern from this issue, but it lands at the same moment.

ci: let a push to the base branch finish instead of cancelling it (#1…

PR #13541 (merged to Dev_new_gui) references this issue with a close keyword.

ci: let a push to the base branch finish instead of cancelling it (#13432)

If this issue is fully resolved, close it manually. If work remains, no action is needed.

`cancelled` base runs still appear after this landed — that is expected, and not a regression

Worth recording, because the obvious reading is that the fix did not work.

After #13541, base ci.yml runs still show up as cancelled when several PRs merge in quick succession. Three merges landed within 8 seconds and produced:

30940902725 pending    18:55:45
30940177405 cancelled  18:46:19   <- 0 jobs
30940172529 cancelled  18:46:15   <- 0 jobs
30940166913 queued     18:46:11

Both cancelled runs have zero jobs:

$ gh api repos/.../actions/runs/30940177405/jobs --jq '[.jobs[] | .status] | ...'
(empty)
$ gh api repos/.../actions/runs/30940177405 --jq '.event'
push

They never started a job. GitHub keeps at most one pending run per concurrency group: when a third arrives while one is running and one is pending, the pending one is cancelled and replaced. That is supersede behaviour, independent of cancel-in-progress.

So the distinction that matters:

before #13541	after #13541
a running base suite was killed mid-flight by the next merge — 23 of 38 base runs cancelled, none succeeded	a running base suite always finishes; only superseded pending runs are dropped

Dropping a pending run is harmless here. The suite measures the base branch, and the newest run measures the newest base — a per-commit measurement is not something anything depends on. The first uncancelled base run after this landed produced the measurement #13162 had been unable to take at all.

How to tell the two apart when diagnosing: check the job count. A cancelled run with zero jobs was superseded while pending. A cancelled run with jobs in completed/in_progress was killed mid-flight, which is the failure this issue was about.

Delivered and verified in the merged tree — closing.

Closes #13432 did not fire because the PR merged to Dev_new_gui, which is not the repository default branch.

Evidence — .github/workflows/ci.yml:39-52 now scopes cancellation to pull requests:

concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  # #13432: cancel superseded PR runs, but let a push to a base branch finish.

The measurement that motivated it is recorded inline: of the last 39 completed base runs, 26 were cancelled and none succeeded — the base branch was effectively never verified.

Why cancelled base runs still appear, and why that is not a regression: runs started before this landed keep their original semantics, and a newly queued run still cancels a pending predecessor (GitHub keeps at most one in-progress plus one pending per group). Base runs now serialise and each in-progress run completes. Dropping cancel-in-progress: true is the whole fix; it does not queue unboundedly.

This restores the post-merge signal that #13162 needs in order to be measurable at all.

fix(ci): cancel superseded pull-request runs in 12 workflows (#14434) (…

Navigation Menu

Repository navigation

ci: cancel-in-progress voids the base-branch suite — no post-merge verification on Dev_new_gui #13432

The base branch is essentially never verified

Measured

Why this blocks #13162

Proposed fix

The trade-off, stated honestly

Acceptance criteria

Provenance

Correction: this does not block the #13162 required-check change

What this issue actually costs — still worth fixing

A correction to the proposed fix, too

Separate throughput note for #13162

`cancelled` base runs still appear after this landed — that is expected, and not a regression

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

ci: cancel-in-progress voids the base-branch suite — no post-merge verification on Dev_new_gui #13432

Description

The base branch is essentially never verified

Measured

Why this blocks #13162

Proposed fix

The trade-off, stated honestly

Acceptance criteria

Provenance

Activity

mrveiss commented on Aug 4, 2026

Correction: this does not block the #13162 required-check change

What this issue actually costs — still worth fixing

A correction to the proposed fix, too

Separate throughput note for #13162

github-actions commented on Aug 4, 2026

mrveiss commented on Aug 4, 2026

cancelled base runs still appear after this landed — that is expected, and not a regression

mrveiss commented on Aug 4, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

`cancelled` base runs still appear after this landed — that is expected, and not a regression