tooling: merge decisions read a status histogram, which cannot see a required context that never ran

The defect

Every merge decision in this fleet today was made by counting check states — pass=N pending=M fail=K. That histogram cannot distinguish a green PR from one where CI never started, because a context that never ran contributes zero to every bucket.

Measured, on a real PR:

#15962   pass=7  pending=0  fail=0        <- reads as green
         10 of 10 REQUIRED CONTEXTS NEVER REPORTED

The seven passes were unrelated checks. No python-suite shard had run. A pending=0 fail=0 read says mergeable; the correct read is not started.

CLAUDE_REVIEW.md already states the rule — "blocked if a required context never reported at all. Count reported contexts" — but nothing implements it, so the rule is prose and the practice is a histogram. That is the same gap as #15900's: documented, and still invisible, because nothing enumerates it.

Two further traps, both hit while measuring this

1. A superseded cancelled is not a failure. Required checks re-run on the same head. Taking tail -1 of an unordered check-runs response reports the stale state:

#15988  17:17:39 cancelled
        17:19:16 cancelled
        17:19:30 cancelled
        17:35:44 success      <- the actual verdict

Not a wrong reading of the data — a right reading of stale data, which is harder to notice. The fix is to take the latest run per name by started_at.

2. Some required contexts are legacy commit statuses, not check-runs. Reading only /check-runs misses them and reports them as never-reported. The two APIs must be unioned.

What a correct check looks like

A working implementation exists and has been run against 18 merged PRs (all clean on it). Its shape:

read the required-contexts list from branch protection — the only thing that knows what should be there
union /commits/{sha}/check-runs with /commits/{sha}/status
take the latest run per context name by start time
report never-reported and not-green separately

That last point is the design decision worth keeping: one means CI has not started, the other means it has and disagrees with you. A single BLOCKED verdict conflates a PR that needs waiting with one that needs work, and those get opposite responses.

This is the same shape as a reach floor bound to what was examined rather than what was found. A histogram counts what reported; only the required-contexts list knows what is missing.

Acceptance criteria

A shared helper answers "is this PR mergeable on required contexts", used by whatever automation merges
It distinguishes never-reported from not-green in its output, not only in its exit code
It unions check-runs with legacy commit statuses, and a test pins a required context of each kind
It takes the latest run per name, with a test that a superseded cancelled followed by a success reads as green
Established by running it against merged PRs known to have had re-runs — reading the code cannot show the stale-state bug
The population of required contexts is read from branch protection, not listed in the tool; a hardcoded list is one protection change from being wrong

docs/developer/CLAUDE_REVIEW.md states the rule this implements
guards: the pre-push pytest gate goes INERT on large changesets instead of failing #15985 — the pre-push gate's WARN … skipping exiting zero: same family, where "did not run" is spelled like "ran and passed"
guard(ci): the filter-coverage checker ignores glob-declared reads, so its green over-claims #15900 — documented and still invisible, because prose is not a population

chore(ci): re-trigger workflows for #15962

merge: resolve base into #15962, and route four converted guards thro…

A live instance, and a cause worth adding: a PR that conflicts with base produces ZERO required contexts.

#15962 sat for over an hour reading pass=7 pending=0 fail=0 with 10 of 10 required contexts never reported. The cause was not a queue, an outage, or a path filter:

not a draft, branch head matched the PR head
ci.yml has no pull_request.paths filter (ci: the backend Python suite gates nothing — python-suite is not a required check #14353 removed it deliberately)
Actions was healthy — two of my other branches triggered normally in the same minute
an empty commit re-triggered CodeQL and semgrep, and still no required context

It conflicted with base. GitHub could not compute the merge commit that pull_request workflows test, so it created no runs at all rather than reporting a failure. Only a local git merge --no-commit --no-ff origin/Dev_new_gui showed it. Resolving the conflicts took the count straight to never-reported=1.

This is the sharpest form of the defect this issue is about:

what happened	what a histogram shows
PR conflicts with base; no workflow can run	`pending=0 fail=0`
PR is fully green	`pending=0 fail=0`

A conflicted PR and a green one are the same reading. And mergeStateStatus was UNKNOWN, not DIRTY, so the usual signal did not name it either.

Two additions to the acceptance criteria this suggests:

The check distinguishes "no runs exist for this head" from "runs exist and passed" — the count of required contexts reported is the discriminator, and it is already the fix this issue proposes
When never-reported equals the full required set, the tool says why it might be — conflict with base being the first thing to test, since it produces no runs rather than failing ones

The diagnostic that settled it is one command and belongs in whatever lands:

git merge --no-commit --no-ff origin/<base>   # then --abort

A PR whose CI has produced nothing for an hour is far more likely conflicted than starved.

fix(tooling): separate a running check from a failed one (#15995)

fix(tooling): surface failing checks outside the required list (#15995)

fix(tooling): a pending unrequired check is not an absent one (#15995)

fix(tooling): one suppressed writer, functions under the ceiling, and…

guard(repo): five guards walked into other checkouts of this reposito…

fix(tooling): three review findings — cross-source masking, unpaginat…

feat(tooling): a required-contexts gate that can see an unreported ch…

PR #16001 (merged to Dev_new_gui) references this issue with a close keyword.

feat(tooling): a required-contexts gate that can see an unreported check (#15995)

If this issue is fully resolved, close it manually. If work remains, no action is needed.

fix(tools): the merge gate reported a merged PR as green, and a skip …

fix(tools): the merge gate's blind spots, and naming the ones that re…

Stale-issue sweep (never auto-closed — Dev_new_gui wasn't GitHub's default branch at merge time). Verified against current origin/main; leaving open.

MET: scripts/pr_required_gate.py (355 lines) + scripts/pr_required_gate_test.py — never-reported/not-green reported as separate outputs; check-runs unioned with the legacy /status (with a pagination-cap comment); latest-run-per-name resolved by started_at with a dedicated superseded-cancelled test; required-context population read from branch protection, not hardcoded; empirically run against 5 real merged PRs known to have had re-runs.

NOT MET: AC1 — "used by whatever automation merges." A repo-wide grep (.py, .sh, .md, .yml) for pr_required_gate outside its own files/tests, and separately for it under .claude/, docs/, or any skill file, returns nothing. The tool is built and well-tested but has zero call sites in any merge automation — it exists, but isn't wired in anywhere.

Three instances today of the class this issue names, from three sessions, each a check whose verdict does not mean what its bucket says. The third is mechanically detectable and worth a rule.

#	What the reader saw	What was true	Where the discriminator was
1	`bucket == "fail"` on three checks of #17478	all three were cancelled, zero genuine failures	`conclusion` (`cancelled` vs `failure`) — present in the same API response, collapsed by the bucket
2	`FAILURE` rollup on #17444 and #17439	cancellations I caused enforcing one-at-a-time	same field, same collapse
3	`python-suite shard 8/12` red on #17505	`3186 passed, 16 skipped`, then a `sys.modules` leak guard set exit 1 after the summary	the test summary itself — a green run inside a red check

I reported (1) to the user as three genuine failures and had to retract it. A peer spent diagnosis effort on (2) before establishing they were mine. (3) was diagnosed correctly by a third session only because they read the log body rather than the conclusion.

The third has a discriminator the other two do not: exit code 1 with 0 failed tests is a contradiction on its face. A pytest run that reports N passed, M skipped and no failures, in a job that exits non-zero, cannot be failing because of the code under test. It is failing because something after the summary — a leak guard, a coverage floor, an atexit hook — set the status. That is cheap to detect and it points the reader at the right half of the log immediately.

Suggested addition wherever the merge decision or the CI-diagnosis path reads a red:

When a failing check's log contains a pytest summary with zero failures, say so in the verdict: "exit 1 with 0 failed tests — the failure is post-summary, not a test result". Do not present it as a test failure.
Never render cancelled as FAILURE in a summary a human or agent reads for a merge decision. cancelled, timed_out, action_required and failure are four different instructions to the reader; the bucket makes them one.
A cancelled check means re-run, and the standing "root-cause every red" rule should say so explicitly, or every serialized queue manufactures phantom root-cause hunts. This is the direct cost of the one-PR-at-a-time discipline and it is currently paid by whoever reads the board next.

The general form, which is the same defect in three coats: the cheapest status surface answers "is this not-success", and every reader treats it as "is this broken". The information needed to separate them was present in all three cases and discarded before the reader saw it.

Related: #15869 (cancellations from queue saturation), #17409 (a hook whose timeout produces DID NOT RUN, a fourth non-failure that reads as one).

fix(tooling): wire the required-contexts gate, close its verdict set,…

fix(ai-ml): the dynamic model loads consult the pinned registry, and …

Navigation Menu

Repository navigation

tooling: merge decisions read a status histogram, which cannot see a required context that never ran #15995

The defect

Two further traps, both hit while measuring this

What a correct check looks like

Acceptance criteria

Related

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

tooling: merge decisions read a status histogram, which cannot see a required context that never ran #15995

Description

The defect

Two further traps, both hit while measuring this

What a correct check looks like

Acceptance criteria

Related

Activity

mrveiss commented on Sep 7, 2026

github-actions commented on Sep 8, 2026

mrveiss commented on Sep 12, 2026

mrveiss commented on Sep 25, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions