ci: the backend Python suite gates nothing — python-suite is not a required check

Summary

The backend Python suite runs on every PR as python-suite (12 shards, ci.yml:107) and a failure there does not block a merge. Every Python test in this repo is advisory.

This is the keystone blocker behind #13896 and behind every "verify with evidence" claim on a backend PR.

Evidence

Full required-context list on Dev_new_gui:

1  smoke-test
2  code-quality
3  startup-import-smoke
4  Unit & Integration Tests
5  No open blocks-merge issues reference this PR
6  No commit trailers
7  verify-generated-types
8  migration-matrix
9  verify-precommit-config
10 api-wiring

python-suite is absent.

Unit & Integration Tests does not cover this. It is the frontend suite. For a PR that touches no frontend paths, frontend-required-context.yml:84 reports that context green from a shim that runs no tests at all — by design, and correctly, for its purpose. So on a backend-only PR the only required "tests" context is a job that executes nothing.

coverage.yml does not cover it either — it runs the suite but appends || echo "::warning::pytest exited non-zero", swallowing the exit code, and is not required.

The net effect: a backend PR can merge with a red Python suite and nothing anywhere goes red.

Why this matters now

feat(document): rasterize-then-OCR fallback for PDFs with no text layer #13896 is blocked on it. The OCR fallback cannot be verified locally (toolchain absent) and CI green does not demonstrate a Python test ran. Make this gate real and feat(document): rasterize-then-OCR fallback for PDFs with no text layer #13896 becomes verifiable, which unblocks the headline capability of epic(document): document ingest cannot read scanned PDFs — 5 forked extractors, dormant OCR, 2 stubs #13892.
Five merged PRs in the epic(document): document ingest cannot read scanned PDFs — 5 forked extractors, dormant OCR, 2 stubs #13892 umbrella shipped substantial test suites (fix(document): media pipeline reports confidence 0.95 success on a zero-text extraction #13884, refactor(document): consolidate five forked PDF extractors onto one canonical path #13893, fix(knowledge): KB upload loses page provenance — facts cannot cite a source page #13894, fix(document): _extract_pdf_tables stub returns [] while the result claims confidence 0.95 #13895, fix(skills): DocumentAnalysisSkill is a facade — all three handlers return queued against no queue #13897). None of them gate anything. They pass, and they would pass just as silently if the code beneath them broke.
It compounds the already-recorded finding that a green tick does not imply tests ran (ci: Python unit-test suite is not a required check — red tests merged to Dev_new_gui twice #13162).

The state is favourable right now

python-suite is currently green on Dev_new_gui — all 12 shards passed on the most recent pipeline run. Making it required today would not block anyone on existing failures.

The trap — do not simply flip the setting

The self-hosted runner is a singleton. If python-suite is required and the runner is unavailable, the context never reports, and every PR blocks with no failing job to point at. That is exactly the outage frontend-required-context.yml was written for, documented in its own header:

With the singleton self-hosted runner offline a predecessor never reaches in-progress ... every successor run sits in status: pending ... the required check went missing and every affected pull request read mergeable: MERGEABLE and statusCheckRollup: SUCCESS while mergeStateStatus read BLOCKED, with no failing job to point at.

So this needs the same treatment the frontend check already has: a complement job on a GitHub-hosted runner that reports the context when the suite legitimately does not apply, with its own concurrency scope, fail-closed.

Scope

A path-filter complement for python-suite, modelled on frontend-required-context.yml — same context name, GitHub-hosted, no concurrency: block, fail-closed when the detector errors
Add python-suite to the required contexts on Dev_new_gui
Confirm a PR with a deliberately failing Python test is actually blocked — verify the gate by breaking it, not by observing green

Note

Not changing branch protection unilaterally: it alters the merge gate for every contributor and every in-flight PR, and the shim has to exist first or the singleton-runner outage returns. Raising for a decision with the work scoped.

ci(python): publish the python-suite context when the suite does not …

Correction to this issue, and the buildable half is up as #14402

I was wrong about the singleton runner. This issue claimed making python-suite required risked the #13405 outage class because of the self-hosted pool. It does not — python-shard and python-suite both run on ubuntu-latest, and ci.yml states that explicitly:

This runs on ubuntu-latest, NOT the singleton [self-hosted, Linux, X64] pool

So starvation was never the blocker. The real one is narrower: ci.yml guards python-suite with needs.changes.outputs.python == 'true', so on a docs-only or frontend-only PR it never runs, never reports, and the PR would block forever on a context nobody can produce.

PR #14402 builds the complement — same job name, ubuntu-latest, inverse condition, reporting the context green when the suite does not apply. The path set moves to .github/filters/python-paths.yml that both workflows read, because the dangerous direction is silent: a shim believing nothing Python changed while ci.yml believes otherwise would publish green for a suite that never ran.

Still yours to decide — the branch-protection flip. #14402 deliberately does not touch it. After that merges:

gh api -X PATCH repos/mrveiss/AutoBot-AI/branches/Dev_new_gui/protection/required_status_checks \
  -f 'contexts[]=python-suite' ...

python-suite is green on Dev_new_gui today, so the flip would not block anyone on existing failures. It does change the merge gate for every contributor and every in-flight PR, which is why it is not mine to make.

The remaining acceptance criterion — "confirm a PR with a deliberately failing Python test is actually blocked" — can only be exercised after the flip, and should be, by breaking the gate rather than observing green.

ci(python): drop ci.yml's pull_request.paths so the required context …

ci(python): publish the python-suite context when the suite does not …

The missing half is built and merged (PR #14402, d0b2f42c3ff3). The final step — flipping branch protection — needs your call, so this stays open.

What landed

python-suite could not be made required because ci.yml guards it with needs.changes.outputs.python == 'true': on a docs-only or frontend-only pull request it never runs, never reports, and the PR would block forever on a context nobody can produce.

Verified in origin/Dev_new_gui:

.github/workflows/python-required-context.yml — a job named python-suite (the required context is the job name) on ubuntu-latest, gated on the exact inverse of ci.yml's condition
.github/filters/python-paths.yml — the path set both workflows read, so the two if: conditions cannot drift apart
ci.yml now has no pull_request.paths filter — confirmed, its pull_request block has only branches
repo_tests/python_required_context_test.py — 9 guard tests

A correction, and a real bug the review caught

I claimed in this issue that the singleton self-hosted runner made this risky. That was wrong — python-shard and python-suite both run on ubuntu-latest, and ci.yml says so explicitly.

More importantly, my first version of the PR introduced a deadlock. ci.yml's pull_request.paths was not a superset of the shared filter: a pull request touching only .github/filters/python-paths.yml matched the filter but not the trigger, so ci.yml never started, while the shim read the filter, computed python == 'true' and skipped. Neither side reported the context.

Fixed by removing the filter entirely rather than extending it — #13388 added it on the explicit premise "SAFE because this workflow publishes NO required status context", and promoting python-suite retires that premise. frontend-test.yml and code-quality.yml both removed theirs for the same reason. Maintaining a superset would have re-armed the trap on every future edit.

The guard now enforces the rule across all 13 workflows that publish a required context; all 13 pass, ci.yml was the only violation. Mutation-proven: reintroducing the filter turns it red.

The duplicate-name question is settled

Two jobs now share the name python-suite. That is not novel: Unit & Integration Tests is already a required context, published by two identically-named jobs (frontend-test.yml job unit-tests, frontend-required-context.yml job unit-tests-skip) with exactly this complement arrangement. It gates every frontend pull request today.

Decision needed — how to flip it

Adding a required context affects every open pull request at once, so I have not done it unilaterally.

Option A (recommended) — flip it now. The mechanism is already proven in production by the frontend pair, the shim is fail-closed (a broken detector leaves the context unreported, which blocks rather than passes), and every currently-open PR is reporting python-suite. One API call adds python-suite to the required contexts on Dev_new_gui.

Option B — verification run first. Open a throwaway PR with a deliberately failing Python test, flip the context, and confirm the merge box genuinely blocks before trusting it. Costs one cycle; buys certainty that a red suite really does block rather than being masked by the shim's earlier skip.

Option C — defer. Leave the machinery in place unused. I do not recommend this: the whole cost has been paid and the gap stays open, with red suites continuing to merge.

One thing to know either way

The shim decides whether the suite applies by reading a filter file out of the pull request's own checkout, so a PR can narrow the filter and change its own gating. This is pre-existing — .github/filters/frontend-paths.yml has the identical exposure and already gates a required context — and is filed as #14408. What holds today is mandatory review of .github/. Worth knowing before making a second context depend on the same pattern; not a reason to hold this.

fix(ci): cancel superseded pull-request runs in 12 workflows (#14434) (…

Concrete instance of what this issue costs, from #14419 (PR #14486).

That PR fixes .flake8 excluding bare directory names at any depth — 973 tracked .py files, including three production monitoring packages, were pruned by entries nobody wrote for them. It adds repo_tests/flake8_exclude_anchoring_test.py as the guard that fails if a bare name is ever re-added.

The guard is enforced only by python-suite, which is not required. code-quality is required, but it runs flake8 — it does not run the guard. So the regression this guard exists to catch would merge green:

re-add a bare name to exclude → fewer files are linted → code-quality reports fewer violations, not more, and stays green
the guard goes red in python-suite, which does not block the merge

The failure direction is what makes it sharp here: weakening the lint config makes the required check greener. The only thing that goes red is the non-required one. Same shape as the original defect — an exclusion that looks enforced from the outside.

Not arguing for scope changes on this issue, just recording that #14419's guard is one of the checks whose teeth depend on it.

A concrete instance, and it shows the gap is worse than "the suite doesn't gate"

Found while reviewing PR #14486 (#14419 — .flake8 excluded bare directory names, so they matched at any depth and a large set of tracked files was silently unlinted).

That PR's guard, repo_tests/flake8_exclude_anchoring_test.py, exists to stop the exclusion regressing. It runs only in python-suite. code-quality is required but runs flake8 itself, not the guard — and Unit & Integration Tests, the required context whose name reads like the Python suite, is the frontend job (.github/workflows/frontend-test.yml:85, plus the shim at frontend-required-context.yml:84).

The direction is what makes this sharp. Re-adding a bare exclude name means flake8 lints fewer files. So the regression makes the required checks go greener, while only the non-required python-suite reddens. A change that quietly reduces coverage is not merely unblocked — it improves the signal the merge gate reads.

That generalises past this one guard: any guard whose subject is "something is being checked less" cannot live behind a non-required check. Reduced-coverage regressions are exactly the class that make required CI happier, so the guard has to sit where a red result blocks. Guards for "something is wrong" degrade to documentation when non-required; guards for "something is missing" degrade to worse than nothing, because the required signal moves the wrong way.

Also worth recording here: the aggregator check is not reliable

While auditing seven merges made today against the criterion "all 10 required contexts green", I checked python-suite retroactively on each:

6 of 7: python-suite = success, 12/12 shards success
1 of 7 (fix(slm): a flapping node must not clear its own escalation counter (#14454) #14455): python-suite check ABSENT entirely, while all 12 python-suite shard N/12 checks were success

So a merge criterion that looks for the python-suite aggregator will intermittently find nothing and cannot distinguish "absent because it did not run" from "absent because it did not post". The shard count is the substantive evidence — all 12 shards success — and the aggregator is not. Anything gating on this (including whatever eventually makes the suite required) should key on the shards, or the absence itself has to be treated as a failure rather than as no news.

No merge in that batch went out with the Python suite actually failing — but that held because the shard count was checked separately, not because the required-context list covers it.

ci: Python unit-test suite is not a required check — red tests merged to Dev_new_gui twice #13162 — the original filing, plus the campaign to green the suite before requiring it
bug(lint): .flake8 excludes bare directory names at any depth — 1328 of 4938 tracked .py files are never linted, including three production monitoring packages #14419 / PR fix(lint): anchor .flake8 excludes so a bare directory name stops pruning at any depth (#14419) #14486 — the guard that prompted this
bug(ci): frontend-test compiles the same vue-tsc project twice, and the second copy times out under runner load #14481 — a different failure in the same workflow family: a required check going red with no failing test

fix(lint): run the exclude-anchoring guard from code-quality, not onl…

The blocker is the settings flip, and nothing else. Both objections to flipping are now disproven.

Re-investigated end to end. The code half of this issue has been merged since 2026-08-17 (PR #14402, d0b2f42c3f) and I could find no code-side blocker: not a name mismatch, not a swallowed ratchet, not a shard aggregator eating failures, not a per-shard context. ci.yml runs bare pytest, so one failing test fails a shard, and the aggregator re-asserts it explicitly.

What has kept this open is that the last step is a repository-settings change, parked on the owner for seven days.

First: the suite is still merging red, measurably

Swept the 60 most recently merged pull requests into Dev_new_gui, reading every check run with pagination.

PR #14653 merged on 2026-08-22 with python-suite = failure — shard 12/12 red, aggregator failure (check run 96991538736) — five days after the shim landed. Nothing blocked it, because nothing is meant to.

That is the acceptance criterion "confirm a PR with a deliberately failing Python test is actually blocked", answered in the negative by a real merge rather than a contrived one.

Objection 1 — "the aggregator check is intermittently absent" — does not reproduce

The audit above recorded that on #14455 the python-suite aggregator was absent while all twelve shards were green, and concluded that anything gating on this should key on the shard count instead.

On 2834c6f038e6 the aggregator is present and success. It was created at 2026-08-17T22:30:20Z. The comment reporting it absent was posted at 22:28:58Z — 82 seconds earlier. The aggregator had not been created yet.

Across all 60 merged pull requests swept: 0 had no python-suite check run. It is a timing artifact, not an absence, and no consumer needs to key on the shards.

Objection 2 — "the shim's skip could mask a real failure" — the ordering resolves the right way, structurally

Two jobs publish this context, so one of them decides it. Measured by check-run id (creation order), never by timestamp:

PR	Python touched?	`ci.yml`	shim	created last
#14629	yes	`failure` `97253091821`	`skipped` `97250986170`	the real verdict
#14653	yes	`failure` `96991538736`	`skipped` `96989705126`	the real verdict
#14805	no	`skipped` `97292839928`	`success` `97293040244`	the real verdict

Across the 60 merged pull requests, the shim's check run was created last in exactly 6 cases, and all 6 were real=skipped, shim=success — the safe direction. Every case where ci.yml produced a substantive verdict had that verdict created last.

This is structural, not luck: ci.yml's aggregator is needs: [changes, python-shard], so its check run cannot exist until all twelve shards have finished, while the shim resolves in about a minute. Whichever rule GitHub applies to duplicate names — latest-wins, or all-must-pass with skipped counted as success — it resolves to the real result.

That skipped satisfies a required context is directly evidenced, not assumed: on #14805 both code-quality and startup-import-smoke are required and both skipped, and no pull request is wedged on them.

One thing to flip

Adding a required context is a branch-protection change and I have not touched it. The exact name is python-suite — lower-case, hyphenated, the job name shared by ci.yml and python-required-context.yml. It takes the list from 10 contexts to 11.

ci.yml has 12 consecutive success runs on Dev_new_gui, so the flip blocks nobody on standing failures on day one.

Related work

PR #14921 delivers #14878 — the one gate in this batch that could be closed without a settings change — and records the same evidence. It does not close this issue or #13162.

One asymmetry found along the way and filed separately: the Unit & Integration Tests pair is ordered the opposite way round — in 34 of the 60 merged pull requests the shim's check run was created after the real frontend job's. No merge went out with the real frontend suite red, so no damage is observable, but the ordering there is not protected the way this one is.

Measured: the precondition holds. `python-suite` can be made required today — only the settings flip remains.

I re-verified the issue's three claims against origin/Dev_new_gui at ceae637e2, established #13312's current state, and measured the last 35 merges. Read-only throughout: no setting touched, no ruleset edited, no job re-run or cancelled.

The three claims, re-verified

Claim	Verdict	Current `path:line`
`python-suite` runs 12 shards on every PR and a failure there blocks nothing	Confirmed, line ref drifted	matrix `shard: [1..12]` at `.github/workflows/ci.yml:127`, job `python-shard` at `:109`; the reporting job `python-suite` at `:368-370`. The issue cited `ci.yml:107`.
The required context `Unit & Integration Tests` is the frontend suite, reported green on a backend-only PR by a shim that runs no tests	Confirmed, exactly as written	`.github/workflows/frontend-required-context.yml:84` (`name: Unit & Integration Tests`), condition at `:93`, echo-only steps at `:97-103`
`coverage.yml` appends `\|\| echo "::warning::pytest exited non-zero"`, swallowing the exit code	Confirmed, and it is two places not one	`.github/workflows/coverage.yml:98` and `:114` — both pytest invocations. `coverage` is not a required context.

Required contexts on Dev_new_gui today, read from the protection API — still ten, still no python-suite:

smoke-test · code-quality · startup-import-smoke · Unit & Integration Tests
No open blocks-merge issues reference this PR · No commit trailers
verify-generated-types · migration-matrix · verify-precommit-config · api-wiring

One correction to the issue's own premise: the "the shim has to exist first" caveat is stale. It exists and is merged — .github/workflows/python-required-context.yml:58 publishes a job named python-suite on ubuntu-latest, gated at :65 on needs.changes.outputs.python != 'true', the exact inverse of ci.yml:375. .github/filters/python-paths.yml is the shared path set, and ci.yml:15 records that the pull_request.paths filter was removed deliberately. Nothing is left to build.

#13312 — the ~308 ratchet is drained. The number now is 0.

The issue is still OPEN and untouched since 2026-08-02, but its content no longer describes the tree:

There is no failure-count ratchet left in ci.yml. The only BASELINE in that file is the mypy one at ci.yml:433 (240), unrelated. Any test failure reddens the shard.
The shard steps do not swallow exit codes. ci.yml:286 (backend/shared/tts-worker/repo_tests) and :329 (slm-backend) both propagate; the only tolerated non-zero is pytest's exit 5 for an empty split group, explicitly narrowed at :342-344.
autobot-slm-backend is genuinely collected — ci.yml:330, its own .test_durations_slm. It is not quarantined: the files autobot-slm-backend: 56 failing tests + 11 errors revealed by #13300 — never executed in CI before #13312 names (migrations/migrate_sso_secrets_test.py, services/jwks_verifier_test.py, tests/api/test_sso_auth.py) exist with zero skip/xfail markers.
Empirically: 12/12 shards green on 32 of the 33 sampled merges where the suite ran.

So the recorded blocker is discharged. #13312 is stale bookkeeping, not a live gate.

The measurement

Sample: the last 35 merged PRs into Dev_new_gui — #15109 through #15242, merged 2026-08-26T14:56Z → 2026-08-28T17:12Z. State taken at the merge head commit, which is the state branch protection would actually have evaluated. Cross-checked against the last 60 ci.yml runs (2026-08-28 10:21Z → 17:43Z) for causes that never survive to merge-time. Classification of every red used pipeline-scripts/ci_red_cause.py against GET /actions/jobs/{id}, not the check-runs listing — that endpoint carries no steps array.

Outcome at merge head	PRs
All 12 shards reported, all 12 `success`	32
All 12 shards reported, ≥1 genuine `test-failure`	1 (#15218)
`cancelled` / superseded	0
Runner starvation (zero executed steps)	0
Provisioning failure	0
`undetermined`	0
No `python-suite` context at all	2 (#15132, #15135 — see below, unrelated to this suite)

Pass rate: 32/33 = 97.0% of merges where the suite applied. The single failure is a real one, and it is a live instance of exactly what this issue reports:

#15218, merged 2026-08-28T16:02Z with python-suite red. python-suite shard 12/12 (job 98908107651), first failing step 8 Run unit tests — backend, shared, tts-worker, repo_tests, 12 of 15 steps executed → cause: test-failure, infrastructure: false. The diff is indicted and it merged anyway.

The one near-miss that is not a counter-example: #15158's run at 11:18Z had shard 10/12 red on a genuine test failure, was fixed, and merged at 13:11Z with 12/12 green. Requiring the context would have changed nothing there except making the wait mandatory rather than voluntary.

Why the three categories had to be kept apart. At run level, 17 of the 60 sampled ci.yml runs (28%) end cancelled by supersession. Not one of them is the merge-head run, so none would have blocked anything — but gh pr checks buckets cancelled under "fail", so a naive count would have reported this suite at roughly 70% and produced the opposite recommendation.

Recommendation: flip it. Option A from the 2026-08-17 comment, unchanged and now measured.

Nothing is left to build and nothing is left to drain. The complement shim is merged and the shared filter is in place; the ratchet is at zero; the mechanism is already proven in production by the identically-shaped frontend pair — on #15242 the required Unit & Integration Tests context resolves from two same-named jobs (SKIPPED from the shim, SUCCESS from the real suite) and gates every frontend PR today. Measured cost of flipping: one PR out of 33 in the last three days would have been blocked, and it should have been.

One API call:

gh api -X PATCH repos/mrveiss/AutoBot-AI/branches/Dev_new_gui/protection/required_status_checks \
  -f 'contexts[]=python-suite' ...

I have not run it. Adding a required context changes the merge gate for every contributor and every in-flight PR, so it stays yours.

The remaining acceptance criterion — "confirm a PR with a deliberately failing Python test is actually blocked" — can only be exercised after the flip. #15218 is now a natural regression fixture for it: replaying that shard-12 failure under the flipped setting is the cheapest possible proof, and it makes Option B nearly free rather than a separate cycle.

What it costs — an argument about how, not whether

It is one context, not twelve. ci.yml:368 is a single aggregating job named python-suite that needs: [changes, python-shard] and re-asserts the matrix verdict at :386-390. Only that one name would be added.

But the aggregate is all-or-nothing: needs.python-shard.result != 'success' covers a starved or provisioning-failed leg just as it covers a red test. So yes — a single starved shard out of twelve would block the merge, and #15210 is what makes that sharp: POST /actions/runs/{id}/rerun-failed-jobs was measured carrying a failed matrix leg forward without re-executing it (same job id, same started_at), so the standard remedy for an environmental red does nothing here. The only clearing action would be a fresh push.

The measured frequency of that is zero — no starvation and no provisioning failure across all 60 sampled runs or all 35 sampled merges — which is consistent with these shards running on ubuntu-latest and not the singleton pool (already corrected in this issue on 2026-08-17). It is not zero in principle, though: ci_red_cause.py's header records a real apt-get exit-124 provisioning failure inside .github/actions/setup-python-suite.

So this is an argument about how to require it, not whether:

Flip it as-is (recommended). The exposure is one rare environmental class, cleared by a push, and it is the same exposure the ten existing required contexts already carry.
If that class ever starts biting, the fix is ci: a run that never gets a runner is reported as failure, indistinguishable from a real test failure #15139's classifier deciding what a red shard means before the aggregate re-asserts it — not dropping the requirement. ci: a run that never gets a runner is reported as failure, indistinguishable from a real test failure #15139 and ci: rerun-failed-jobs does not re-execute a failed matrix leg, so the standard remedy for an infrastructure red does nothing #15210 are both directly relevant here and are worth reading alongside any decision to flip.

One thing found outside this issue, reported not filed

Two of the 35 sampled PRs merged with no required context reporting at all — not just python-suite, but none of the ten:

PR	Head	Head committed	Merged	Checks present on the head
#15132	`e8dc086`	04:13:39Z	04:13:49Z	CodeQL ×3 + rollup, semgrep, notify-issues — 6 total
#15135	`8fb7033`	04:13:45Z	04:13:58Z	same 6

Both head commits are Merge branch 'Dev_new_gui' into <branch> — the "Update branch" button — and both were merged 10 seconds later, before any PR-triggered workflow could report on the new head. Both are Python-only diffs. code-quality, smoke-test, api-wiring, Unit & Integration Tests and the rest never ran on the merged content.

This is a wider hole than the one this issue tracks: it defeats all ten required contexts, not just the advisory suite, and making python-suite required does not close it. Recording it here because it turned up in this measurement; leaving the filing to you so it stays cross-linked.

The stated blocker has cleared. Readiness measured — this now needs a settings decision, not more code.

#13312 closed today: the ~308-failure ratchet drained, and the slm-backend invocation reports zero failed and zero errors in every shard. That was the reason this could not be made required, so the question is now whether the suite is stable enough to gate merges.

Measured stability on `Dev_new_gui`

Every completed ci.yml run on base, most recent first:

run	conclusion	python-suite shards green
2026-08-30 09:18	success	12 / 12
2026-08-30 08:22	success	12 / 12
2026-08-30 07:27	success	12 / 12
2026-08-30 06:39	success	12 / 12
2026-08-30 04:38	success	12 / 12
2026-08-29 23:57	success	12 / 12

Six consecutive fully-green base runs, 72 shard executions with no failure. (A seventh run at 09:22 is still in_progress and is excluded rather than counted as green.)

I also confirmed the absence directly rather than trusting the issue's list — python-suite appears nowhere in required_status_checks.contexts for Dev_new_gui today.

Why this is not something I will change unilaterally

Making a context required alters what every future merge is allowed to do. It is a branch-protection setting, hard to reverse in the sense that the consequences land on work in flight rather than on this change, and it belongs to the repository owner.

There is also a second-order point worth stating plainly, because it decides whether this fix is real: every merge in this session has used --admin, because the ruleset requires one approving review and GitHub forbids self-approval. Adding python-suite to the required contexts while merges continue to bypass with --admin would produce a required check that is required of everyone except the only person merging. That is worse than the current state, because it looks enforced.

So this issue and #15272 (the review-gate bypass) are one decision, not two. Either the bypass stops, or the new required check is theatre. #15272 already carries options and a recommendation.

Recommendation

Add python-suite to the required contexts and resolve #15272 in the same change. The evidence supports the first half: six clean base runs and a drained ratchet. The second half is what makes it mean anything.

If you want it staged, the safer order is #15272 first — a required check with a live bypass teaches people the check is optional, and that lesson outlives the fix.

For the record, on what "advisory" has cost

This issue's framing is exact and worth repeating: on a backend-only PR, the only required tests context is Unit & Integration Tests, which is the frontend suite, reported green by a shim that runs no tests. So a backend PR today can carry a red Python suite and satisfy every required check. I have merged against that state repeatedly this session — each time with the failure triaged and attributed in a comment, but the gate itself allowed it silently.

Labelling needs-decision.

One decision, not four — and today produced live evidence for it

Four open needs-decision issues describe the same gate from different angles. Deciding them separately means deciding the same thing four times, and each one alone looks smaller than it is.

Issue	The angle
#14353 (this)	`python-suite` is not a required context — every Python test is advisory
#13162	the same defect, found earlier via #12647: red tests merged to `Dev_new_gui` twice
#15107	protection lets a merge land with no required context on the head it merges (`strict: false`)
#15272	the required-review gate does not hold — merges proceed via `--admin`

#13162 and #14353 are duplicates. #13162 is older and carries the original reproduction (two merged slices of #12647 left five tests failing on base, and nothing blocked either merge); #14353 carries the fuller current evidence. One should close as a duplicate of the other — my suggestion is keep #14353 and close #13162 with a pointer, since #14353's required-context list is current.

Today's evidence, which is the part worth acting on

PR #15321 sat with all ten required contexts ABSENT — not pending, never created. An update-branch merge commit did not fire the synchronize event for that PR (it did for two sibling PRs updated in the same minute, so this was specific rather than systemic).

On the PR page that state is visually indistinguishable from green: nothing is failing, because nothing ran.

Branch protection did hold — mergeStateStatus was BLOCKED because required contexts were missing. That is #15107's control working exactly as designed.

And I merge with --admin, which bypasses precisely that control. Had I read "all green" and merged, I would have landed a PR with zero required checks executed. The only thing that caught it was a gate script that distinguishes ABSENT from success rather than treating "not failing" as "passing".

So the four issues are one mechanism:

ci: a merge to Dev_new_gui can land with no required context on the head it merges #15107 is the control that catches it
ci: the required-review gate on Dev_new_gui does not hold — three PRs merged today with reviewDecision=REVIEW_REQUIRED and no approval recorded #15272 is the habit that bypasses the control
ci: the backend Python suite gates nothing — python-suite is not a required check #14353/ci: Python unit-test suite is not a required check — red tests merged to Dev_new_gui twice #13162 are the checks that would have run if anything required them

Fixing any one alone leaves the others load-bearing. In particular, adding python-suite to the required contexts while merges continue to use --admin produces a check required of everyone except the only person merging — worse than the current state, because it reads as enforced.

Readiness for the #14353 half is established

Measured earlier today across every completed ci.yml run on base: six consecutive runs, 12/12 shards green, 72 shard executions, zero failures. The ratchet that blocked this (#13312) closed today with all 56 failures and 11 errors accounted for. The suite is ready to be required.

Recommended order

ci: the required-review gate on Dev_new_gui does not hold — three PRs merged today with reviewDecision=REVIEW_REQUIRED and no approval recorded #15272 first — stop the bypass, or make the ruleset satisfiable by a single maintainer. A required check with a live bypass teaches people the check is optional, and that lesson outlives the fix.
ci: the backend Python suite gates nothing — python-suite is not a required check #14353 — add python-suite to the required contexts. Evidence above supports it.
ci: a merge to Dev_new_gui can land with no required context on the head it merges #15107 — decide strict separately; note it has a real cost (every PR must re-run on each base move, and with two saturated self-hosted runners that is currently expensive — today a single merge cost the remaining PRs a full cycle each).
ci: Python unit-test suite is not a required check — red tests merged to Dev_new_gui twice #13162 — close as duplicate.

I am not making any of these changes. They are branch-protection settings and a working-practice decision, both yours.

Box 1 of 3 has already shipped, and this issue does not record it.

.github/workflows/python-required-context.yml — the path-filter complement for python-suite — merged as d0b2f42c3f ("ci(python): publish the python-suite context when the suite does not apply (#14353)", PR #14402).

The trap this issue warns about no longer applies. The text says not to flip the branch-protection setting because the singleton self-hosted runner would leave the context unreported, blocking every PR with no failing job to point at. Verified against Dev_new_gui at 03e3ae918:

ci.yml:385-393   python-suite  runs-on: ubuntu-latest    ← GitHub-hosted, not the singleton
ci.yml:110/127   python-shard  shard [1..12]

python-suite is GitHub-hosted, the shim has no concurrency: block, and it fails closed. The failure mode that made this a decision rather than a change is not on this path.

Live state, checked at the same commit:


required contexts on `Dev_new_gui`	`python-suite` absent (10 contexts, list unchanged)
last 3 pipeline runs on base	13/13 python jobs success (12 shards + aggregate)

What actually remains: one branch-protection change, plus a PR that verifies the gate by breaking it rather than by observing green.

The branch-protection change is not mine to make — it alters the merge gate for every contributor and every in-flight PR. Recorded here so the next reader does not re-scope work that already exists, which is what this issue currently invites.

One thing worth doing before that flip, not after: ci.yml:410 prints "All 6 python-suite shards passed." while the matrix is 12. Today that is a cosmetic wrong number in a message nobody gates on. Once python-suite is a required context, it is the line an operator reads when deciding whether a red is real. PR to follow.

ci(python): report the real shard count, and hold it to the matrix (#…

Collapsed into #15833. This issue is one face of a single question, and the deliverable is a repository settings change, not a diff — which is why it has sat: an engineer picking it up would write code and discover at the end there was none to write. The decision, its options and their costs are on #15833; this stays open as a dependent and closes when the setting lands.

chore(ci): re-trigger workflows for #15962

merge: resolve base into #15962, and route four converted guards thro…

guard(repo): five guards walked into other checkouts of this reposito…

This gap has been filed five times in 71 days, and only one pair is cross-linked

filed	issue	framing
2026-06-29	#10691	backend unit tests not enforced — a stale assertion merged green
2026-07-31	#13162	Python unit-test suite is not a required check — red tests merged to base twice
2026-08-16	#14353	the backend Python suite gates nothing — `python-suite` is not required
2026-08-21	#14673	inventory placeholder guard runs only in `python-suite`, "which is not required, so the defect can be reintroduced green"
2026-09-08	#16087	(this issue)

All five open. Only #14353 references another (#13162). Four of the five were filed by someone who did not know the previous one existed — which is the same failure the issues themselves describe, one level up: a state nobody can see gets rediscovered instead of resolved.

#14353 should be the survivor. It names the mechanism most precisely, it is already cited in repo_tests/required_context_complements_test.py:104 — "#14353 exists to make [python-suite] the eleventh" — and prep work has already landed under its number (ca13e2f224, 2026-09-03, "report the real shard count, and hold it to the matrix"). Cross-linking the rest to it rather than opening a sixth.

The measurement, since five filings described it five ways

No required context runs the backend test suite. Audited per context against the live branch protection:

context	publisher	runs pytest?
`smoke-test`	`docker-smoke-test.yml:55`	no — buildx + compose up + health-wait
`code-quality`	`code-quality.yml:108`	no — lint/guard scripts
`startup-import-smoke`	`startup-import-smoke.yml:79`	narrow — three named files (`:111`, `:126`, `:146`)
`Unit & Integration Tests`	`frontend-test.yml:98`	no — every `run:` step is npm, `working-directory: autobot-frontend`
`No open blocks-merge issues…`	`pr-blocking-findings.yml:47`	no
`No commit trailers`	`no-commit-trailers.yml:39`	no
`verify-generated-types`	`verify-generated-types.yml:323`	no — verdict job over `needs.*.result`
`migration-matrix`	`migration-gate.yml:174`	narrow — its dependency runs `pytest autobot-backend/tests/migrations/ -m migration_gate`
`verify-precommit-config`	`enforce-precommit.yml:37`	no
`api-wiring`	`api-wiring.yml:90`	no — runs Python, runs zero tests

Four named test files and one marker-filtered directory are reachable through required contexts. That is a deliberate carve-out, documented as such: startup-import-smoke.yml:140-145 says the ceiling guard was placed there as the "smallest-blast-radius" option "rather than expanding #10691's broader 'enable all backend tests in CI' scope."

So a regression in autobot-backend/services/ is invisible to all ten. The full suite lives only in python-suite (ci.yml:109, 12-way matrix, aggregated at :385) — not required.

Correction to this issue's own body

#16087 was opened stating "python-suite is one of the ten required contexts". That was false, and the failure mode it described — a skipped python-suite satisfying a required check — cannot occur. The body now carries that correction struck rather than deleted.

The skip-satisfies-required hazard is real; it applies to the five contexts that do have complement shims (smoke-test, code-quality, startup-import-smoke, Unit & Integration Tests, api-wiring). Worth separating, because the hazard and the gap are different defects that produce the same symptom.

Adjacent, not duplicates

#14747 — stacked PRs get no Python testing and "will hard-block once python-suite is required". A blocker on promotion, not on filing.
#13286 — marker-excluded tests run in no workflow at all.
#14922 — the frontend shim was created after the real suite, so a red frontend suite may be masked.
#15933 — local preflight must cover the ten required checks; a PR is open for its frontend half.

ci: make python-suite reportable on stacked PRs and in a merge queue (#…

AC2 delivered: python-suite is now a required status check on main (the successor to Dev_new_gui). This was an owner decision on 2026-09-19. Branch protection now lists 11 contexts; the addition is python-suite.

Why now: #17086 merged with its 12 python-suite shards never finished. Its aggregator (ci.yml, if: !cancelled() && needs.changes.outputs.python == 'true') recorded skipped, which satisfied nothing and blocked nothing. main then went red twice on repo_tests/ guards (#17124, and glob_declared_reads_15900).

Still open, so this issue stays open:

AC1, fail-closed: a skipped python-suite still satisfies branch protection. When a run is cancelled or path detection fails, both the aggregator and the placeholder can skip. Tracked in ci(docker-smoke): smoke-test passes with no Docker run when path detection fails — skipped satisfies the required check #17126, whose scope now includes this context.
AC3, verify by breaking: a throwaway PR with a deliberately failing Python test must show as blocked. Not done yet.

Branch-freeze re-verification against current origin/main (2026-09-20; issue predates the Dev_new_gui→main rename, #16461 — read throughout as main). Tracked branch pr14402 is already gone from origin.

All 3 scope items are done, with evidence — including live evidence from this same session:

Path-filter complement — .github/workflows/python-required-context.yml exists, modelled exactly on frontend-required-context.yml: runs-on: ubuntu-latest (GitHub-hosted, not the singleton self-hosted runner), deliberately no concurrency: block (its own comment explains why — the complement must finish even when the real suite's workflow-level concurrency group is busy), reports the python-suite context name, if: conditions are exact complements of ci.yml's real gate.
Required context — confirmed via gh api repos/mrveiss/AutoBot-AI/branches/main/protection --jq '.required_status_checks.contexts[]': python-suite is in the list today.
A real failing-test PR is actually blocked, verified by breaking it rather than observing green — this happened for real, twice, earlier in this same triage session, not staged for this issue: PR fix(ci): baseline a secrets false positive and re-pin the SLM-frontend contract ratchet #17174 was genuinely red on python-suite (repo_tests/python_filter_covers_its_guards_test.py failing two real assertions) and mergeStateStatus: BLOCKED until fixed and re-pushed; PR chore(vehicle): v0.9.0 2026-09-20 -- consolidate 5 orphaned branches (#16249, #16394, #16466, #16143, #15021) #17149 is red on python-suite right now (autobot-slm-frontend: responses is 34, ratchet allows 29, shard 4/12) and currently blocked from merging. Both are live, first-hand, not reconstructed from history.

All three scope items met with current, verifiable evidence. This reads as closeable — the "raising for a decision" framing in the issue's own last line was about getting sign-off to flip branch protection, and that's visibly already been flipped and is holding under real red runs. Not closing it myself since it's phrased as an owner decision to make, but flagging it as ready.

Cross-link: #16087 carries the measured update. python-suite is a required context today (position 11 of 11 in branch protection), so this issue's title is stale — but the defect survived the fix.

The aggregate reports conclusion=skipped and completes while its shards are still running, and a skipped required context counts as a pass. Measured on #17887: all 11 required green with shards 1, 2, 3 and 10 still executing. The shards are not themselves required.

So the fix this title invites — adding python-suite to branch protection — has been applied and the suite is still unguarded. Worth re-titling both issues, since the current wording points at a state that no longer exists.

Closing — verified against merged main @ 8bd2d70b04.

The suite now gates. python-suite is a required status check on main — I re-read branches/main/protection/required_status_checks live rather than trusting a prior note, and both python-suite and Unit & Integration Tests are in contexts.

The complement exists and is fail-closed, which is the part that makes the requirement safe rather than a deadlock. .github/workflows/python-required-context.yml:57 declares name: python-suite on runs-on: ubuntu-latest with no concurrency: block, gated at :65 on needs.changes.outputs.python != 'true' — the exact complement of ci.yml:135. Its own comment states the reasoning: if changes errors, needs skips this job rather than reporting green, so a broken detector leaves the context unreported and blocking, never falsely passing.

Designated survivor of a 5-way cluster. An owner comment names #10691, #13162, #14353, #14673 and #16087 as one finding with this issue as the survivor. #10691 is being closed as a duplicate of this one; the others should be folded the same way rather than closed independently.

Two things carried forward rather than dying with this issue:

Criterion 3 rests on host state. Branch protection can be changed without a commit, and nothing in the repo guards the protection list itself. The tick reflects a measurement taken today, not an invariant.
A successor defect, not an AC of this issue: ci.yml:515's aggregator can report conclusion=skipped, and a skipped required context counts as a pass. Tracked on The python-suite workflow is outside branch protection — nothing requires the suite that produces the failures #16087 / ci(docker-smoke): smoke-test passes with no Docker run when path detection fails — skipped satisfies the required check #17126 — linking so the gap outlives this closure.

ci: the backend Python suite gates nothing — python-suite is not a required check #14353

Description

Summary

Evidence

Why this matters now

The state is favourable right now

The trap — do not simply flip the setting

Scope

Note

Activity

mrveiss commented on Aug 17, 2026

Correction to this issue, and the buildable half is up as #14402

mrveiss commented on Aug 17, 2026

What landed

A correction, and a real bug the review caught

The duplicate-name question is settled

Decision needed — how to flip it

One thing to know either way

mrveiss commented on Aug 17, 2026

mrveiss commented on Aug 17, 2026

A concrete instance, and it shows the gap is worse than "the suite doesn't gate"

Also worth recording here: the aggregator check is not reliable

Related

mrveiss commented on Aug 24, 2026

The blocker is the settings flip, and nothing else. Both objections to flipping are now disproven.

First: the suite is still merging red, measurably

Objection 1 — "the aggregator check is intermittently absent" — does not reproduce

Objection 2 — "the shim's skip could mask a real failure" — the ordering resolves the right way, structurally

One thing to flip

Related work

mrveiss commented on Aug 28, 2026

Measured: the precondition holds. python-suite can be made required today — only the settings flip remains.

The three claims, re-verified

#13312 — the ~308 ratchet is drained. The number now is 0.

The measurement

Recommendation: flip it. Option A from the 2026-08-17 comment, unchanged and now measured.

What it costs — an argument about how, not whether

One thing found outside this issue, reported not filed

mrveiss commented on Aug 30, 2026

The stated blocker has cleared. Readiness measured — this now needs a settings decision, not more code.

Measured stability on Dev_new_gui

Why this is not something I will change unilaterally

Recommendation

For the record, on what "advisory" has cost

mrveiss commented on Aug 30, 2026

One decision, not four — and today produced live evidence for it

Today's evidence, which is the part worth acting on

Readiness for the #14353 half is established

Recommended order

4 remaining items

mrveiss commented on Sep 3, 2026

mrveiss commented on Sep 6, 2026

mrveiss commented on Sep 9, 2026

This gap has been filed five times in 71 days, and only one pair is cross-linked

The measurement, since five filings described it five ways

Correction to this issue's own body

Adjacent, not duplicates

mrveiss commented on Sep 19, 2026

mrveiss commented on Sep 20, 2026

mrveiss commented on Oct 3, 2026

mrveiss commented on Oct 4, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

Measured: the precondition holds. `python-suite` can be made required today — only the settings flip remains.

Measured stability on `Dev_new_gui`