ci: strict=false means a PR can merge without ever being checked against a guard that landed while it was open

The window

required_status_checks.strict is false on Dev_new_gui, so a PR merges on checks computed against whatever the base was when those checks ran — not against the base it actually merges into. Nothing re-evaluates them in between.

That is normally a mild staleness risk. It becomes a correctness hole the moment a new repo-wide invariant lands, because a PR opened before the guard exists can merge after it without ever being checked against it.

The instance that proved it

Today, within a two-minute span:

Event	UTC
PR #14470's `code-quality` run started (base `20036f964`)	20:36:57
`c1d1dd8cd072` (#14236 / PR #14463) merged — first version of the file-size ceiling numbers	20:38:50
PR #14470 merged as `0bd3326cb07f`	21:44:43

Before c1d1dd8, KNOWN_LARGE was a bare set — an unconditional exemption with no line number. #14470's run checked out that version, so tool_handler.py at 4081 lines was exempt, not violating, and the check passed truthfully. Sixty-five minutes later it merged against a base where 4063 was the recorded ceiling, and Dev_new_gui went red for every open PR (#14495).

Neither the check nor the pre-commit hook did anything wrong. The gap is entirely between "checks computed" and "merge performed".

Why this is worth deciding rather than absorbing

Every guard, ratchet, lint rule and invariant test this repo adds inherits the same window — and the repo adds them constantly. Each one has a period during which already-open PRs can land unchecked, and the resulting breakage lands on Dev_new_gui, where it blocks every open PR at once rather than the one that caused it.

It is also silent. Note the interaction with the base-run cancellation problem (#14434 / #13432): the push run that would have reported the newly-red base was cancelled by the next merge, so the failure did not surface for 29 minutes and 4 merges.

Options — this is a cost/policy call, not a small change

strict: true (require branches up to date before merging). Closes the window completely. Cost: every merge forces every open PR to re-run before it can merge, on a singleton self-hosted runner. With ~8 PRs in flight, that is a serialisation cost measured in hours, not minutes. Probably unaffordable as-is.
A merge queue. Same guarantee, batched — the queue re-tests each candidate against the actual post-merge state and merges only what passes. Better throughput than (1), but it is a real workflow change and still adds runner load.
Targeted re-verification. Leave strict: false, but when a PR that introduces or tightens a repo-wide guard merges, re-trigger CI on all open PRs. Cheap and precise, but it depends on recognising "this PR adds an invariant", which is a judgement call and will be missed.
Accept it and make it loud. Do nothing about the window, but fix base-run cancellation (infrastructure(ci): 12 PR-triggered workflows have no concurrency group — 42 of 271 queued runs were for superseded commits #14434 — in flight now) so a red base surfaces within one merge rather than four, and make the first PR to notice report it rather than silently inherit it.

I recommend 4 now, 2 later: base-run visibility is already being fixed and is nearly free, while a merge queue is the only option that actually closes the window without serialising the runner. 1 is not viable on the current runner.

bug(ci): Dev_new_gui is red — tool_handler.py grew past its #14236 ceiling (4081 vs 4063), failing code-quality on every open PR #14495 — the outage this came out of
tech-debt(lint): the file-size hook's grandfather list has no ceiling — the three exempted files grew to 1.4x-6.8x the limit #14236 / PR fix(lint): give the file-size grandfather list a ceiling that only shrinks (#14236) #14463 — the guard whose landing opened the window
infrastructure(ci): 12 PR-triggered workflows have no concurrency group — 42 of 271 queued runs were for superseded commits #14434, ci: cancel-in-progress voids the base-branch suite — no post-merge verification on Dev_new_gui #13432 — base-branch runs cancelled by the next merge, which is why it stayed invisible
[dev_new_gui_branch_protection] — 2 layers, 10 required contexts, strict = false

A second, sharper instance — the merge ref itself lags the base

The first instance in this issue was a 65-minute gap between checks running and the merge happening. This one is 72 seconds, and it is worse, because nothing about it looks stale.

PR #14476's code-quality run:

created / started : 2026-08-17T23:41:48Z
base fix merged   : 2026-08-17T23:40:36Z   (fb7f85f71176, tool_handler 4081 → 3863)
checked out       : Merge f630861a68a2 into a57e1a449609

The run was created 72 seconds after the fix landed on Dev_new_gui, and still built its merge ref on a57e1a449609 — the previous base. It then failed with:

autobot-backend/chat_workflow/tool_handler.py: 4081 lines, over its recorded ceiling of 4063.

— a breach that no longer existed anywhere in the repo at the moment that job ran.

GitHub recomputes a PR's merge ref asynchronously, and a run created after a base change can still test the pre-change base. So "the run started after the fix" is not evidence the run tested the fix. The only reliable signal is the HEAD is now at … Merge <pr-head> into <base-sha> line in the checkout step — the base SHA it actually used.

Why this compounds the problem already filed here

The window described in this issue is "checks computed against an older base than the one merged into". This instance shows the window has no clean lower bound in wall-clock time — it is not "PRs older than N minutes", it is "any run whose merge ref was resolved before the base moved", and that resolution is invisible unless you read the checkout line.

Two practical consequences:

A close/reopen re-trigger is not guaranteed to be enough on its own. If it fires while the merge ref is still stale, the fresh run tests the old base again. Re-triggering and then confirming the checkout line names the current base is the only way to be sure.
A red check on a PR you have not touched should be diagnosed by its checkout base, not by its timestamp. I nearly recorded this as a real failure of fix(provision): drop deprecated/stale openvino pins across all seven pip sites (#14452, #14453) #14476 — the run was newer than the fix, so by timestamp it looked legitimate.

Re-triggered #14476; the PR itself is unmodified.

Related: #14495 (the outage), #14450 / PR #14501 (base-run cancellation, which is why the red base stayed invisible in the first place).

Close/reopen does NOT refresh a stale merge ref — measured

Correcting my earlier note on this issue. I wrote that a close/reopen re-trigger "is not guaranteed to be enough." It is stronger than that: on two PRs it did nothing at all, twice each.

PRs #14476 and #14488 were closed/reopened at 00:26Z and 00:34Z — well after the base fix merged at 23:40Z. Fresh code-quality runs started at 00:52Z and 00:44Z. Both checked out the same stale merge commit as before:

PR #14476: HEAD is now at 616b080 Merge f630861a68a2 into a57e1a449609
PR #14488: HEAD is now at 06d0493 Merge 8e5890f538fa into a57e1a449609

a57e1a449609 is the pre-fix base. Byte-identical merge SHAs (616b080, 06d0493) to the runs from 90 minutes earlier — GitHub reused the cached merge ref rather than recomputing it. Both failed on tool_handler.py: 4081 lines, a breach that had not existed anywhere in the repo for over an hour.

So a new workflow run is not a new merge base. The pull_request event fires, a fresh run is created, it checks out the stale ref, and the failure looks current — same reasoning that nearly had me record #14476 as a genuine failure earlier.

What actually works

PUT /repos/{owner}/{repo}/pulls/{n}/update-branch — it merges the base into the PR branch, which changes the branch and therefore forces a genuinely new merge ref. Applied to both.

The ordering rule that falls out of this, for anyone triaging a red PR they did not touch:

Read the checkout line — HEAD is now at <sha> Merge <pr-head> into <base-sha>. That base SHA is the only ground truth. Timestamps do not tell you which base was tested; neither does the run being newer than the fix.
If the base SHA is stale, do not close/reopen — it may re-run against the same cached ref and waste a slot on the singleton runner (this cost four runs across two PRs).
Use update-branch, then confirm the next run's checkout line names the current base.

This strengthens the case for a merge queue (option 2 in this issue): a queue re-tests each candidate against the actual post-merge state by construction, rather than depending on GitHub choosing to recompute a ref.

Related: #14495 (the outage), #14450 / PR #14501 (base-run cancellation, which is why the red base went unnoticed for 29 minutes).

A third instance, and the first where it would have merged something untested

PR #14506 is fully green — 10 required contexts, 12/12 shards, zero failures. Those checks ran against a merge base that excludes eeee402ea, the sibling PR I merged an hour earlier for the same issue.

eeee402ea rewrote roughly 120 lines of reconciler.py module scope: RESTART_CHURN_WINDOW_S, _restart_churn_active, and new imports for parse_utc_iso, SERVICE_DISCOVERY_TTL_S and is_managed_autobot_service. The file auto-merges textually, so nothing flags it — but #14506's 781-line test module has never executed against that content.

Green, mergeable-looking, and untested at the state it would actually produce.

It also carries a real textual conflict in autobot_shared/env_registry.py:1684, where both PRs register an env var at the identical insertion point. git merge-tree puts the markers inside a single EnvVarSpec( call, so a careless resolution yields two name= fields or an unterminated call — and env_registry.py failing to import takes every consumer down at boot. That one at least surfaces as mergeStateStatus: DIRTY; the stale-test-coverage half does not surface at all.

Why this instance is worse than the first two. #14476 and #14488 were false red — stale bases producing failures that no longer existed, costing time but never risking a bad merge. This is the opposite: stale green. The earlier ones were caught by reading the checkout line because they failed; nothing prompts you to read it when everything passes.

The practical rule that falls out, and the reason I am still recommending option 2 (merge queue):

When two PRs touch the same file, the second one's green is only meaningful if its merge base includes the first. Check the checkout line's base SHA against the current base before merging, not only when diagnosing a failure.

A merge queue makes that structural — it re-tests each candidate against the actual post-merge state — rather than depending on someone remembering to compare two SHAs while everything looks fine.

Related: #14495 (the outage), #14450 / PR #14501 (base-run cancellation, merged), #14465 (the issue both PRs serve).

Collapsed into #15833. This issue is one face of a single question, and the deliverable is a repository settings change, not a diff — which is why it has sat: an engineer picking it up would write code and discover at the end there was none to write. The decision, its options and their costs are on #15833; this stays open as a dependent and closes when the setting lands.

Relationship to #15833 checked, and it is not recorded anywhere

A triage pass reported this issue as collapsed into #15833. Neither issue says so — #14500's body and comments do not mention #15833, and #15833's body and comments do not mention #14500. I checked both directions before acting, and I am not closing on it.

They are topically adjacent: #15833 asks the general question (what must be true of a branch before it may merge), and this issue is one concrete instance of getting that wrong (strict=false lets a PR merge without being re-checked). That is a plausible reason someone would collapse them, which is exactly why it needs to be recorded rather than inferred.

If the collapse was intended, it should be an explicit supersede note on this issue naming #15833, and then this closes. If it was not, this stays open on its own terms and #15833's answer may resolve it as a side effect. Leaving it open is the reversible choice.

Navigation Menu

Repository navigation

ci: strict=false means a PR can merge without ever being checked against a guard that landed while it was open #14500

The window

The instance that proved it

Why this is worth deciding rather than absorbing

Options — this is a cost/policy call, not a small change

Related

A second, sharper instance — the merge ref itself lags the base

Why this compounds the problem already filed here

Close/reopen does NOT refresh a stale merge ref — measured

What actually works

A third instance, and the first where it would have merged something untested

Relationship to #15833 checked, and it is not recorded anywhere

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

ci: strict=false means a PR can merge without ever being checked against a guard that landed while it was open #14500

Description

The window

The instance that proved it

Why this is worth deciding rather than absorbing

Options — this is a cost/policy call, not a small change

Related

Activity

mrveiss commented on Aug 18, 2026

A second, sharper instance — the merge ref itself lags the base

Why this compounds the problem already filed here

mrveiss commented on Aug 18, 2026

Close/reopen does NOT refresh a stale merge ref — measured

What actually works

mrveiss commented on Aug 18, 2026

A third instance, and the first where it would have merged something untested

mrveiss commented on Sep 6, 2026

mrveiss commented on Oct 3, 2026

Relationship to #15833 checked, and it is not recorded anywhere

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions