Repository navigation
bug(guards): reach_declarations_test passes locally on a commit CI fails — a local pass is not evidence #17144
Description
Problem
On #17072 head ff296b7142, CI fails:
ReachFloorError: [prompt-injection-detector-strict-mode] floor 2940 sits 304 below
its live population of 3244 tracked backend python files (tests excluded),
which exceeds the declared allowance of 300 (skips=0 + growth=300).
FAILED repo_tests/reach_declarations_test.py::test_every_declared_floor_is_pinned_to_its_population[prompt-injection-detector-strict-mode]
Running that same test against that same commit in a detached worktree passes — twice, in two
separate worktrees (224 passed, then 34 passed for the parametrized floor test alone).
The population data is present locally. In the same worktree, the guard's own discovery command
returns the same number CI reports:
git ls-files -- 'autobot-backend/*.py' | grep -v '_test\.py$' | wc -l -> 3244
So the guard had the failing number available and did not fail on it.
Why this matters
A guard that passes locally and fails in CI on identical input is worse than one that simply fails:
it converts local verification into false assurance. Working from that local pass, the obvious
conclusion was "the floor is fine, something else in shard 12 is failing" — which is wrong, and
would have sent someone looking for a second, non-existent defect while the real one sat in plain
sight in the error message.
This also undermines the pre-push hook's value for this class of guard: a developer who runs these
tests before pushing gets a green that does not predict CI.
What is NOT the cause
- Main moving under the branch:
git log 0f5b45cab..origin/mainis empty andbehind-okreports
behind=0. - A stale checkout: the worktree was created fresh from
origin/issue-17043-17056-approval-consolidation
andgit log --oneline -1confirmedff296b7142. - The discovery command failing outright: it returns 3244 in that worktree.
Unresolved
I did not determine why the guard does not act on the number its own discovery returns. Candidates
worth checking, none confirmed: repo_root()'s lru_cache interacting with worktrees;
reach_declarations_test driving discover against a substitute root (its docstring mentions
being driven against an empty directory for #15826) so the real root is never measured; or
scrubbed_git_env() behaving differently under a worktree than under a plain checkout.
Stating that as unresolved rather than picking the most plausible one — a guessed cause here would
send the fix at the wrong mechanism.
Acceptance criteria
- The reach floor guard produces the same verdict locally and in CI for the same commit,
demonstrated onff296b7142(or a reconstruction of it) failing in both. - If the divergence is worktree-specific, the guard either works correctly in a worktree or
refuses to run there with an explicit message — never a silent pass. - A test asserts the guard fails when the population exceeds floor+growth, driven through the
same entry point CI uses, so a future vacuous pass is caught.
Notes
Found landing #17072. Related: #17142 (the same floors re-pinned three times in one night) and
#15928 (floors set below their own discovery).
Activity
fix(migrations): re-parent shared-link-tracking onto #17072 head, re-…
chore(guards): re-pin prompt-injection-detector-strict-mode floor 294…
fix(updater): re-detect ansible_dir per execute_playbook call, not on…
chore(vehicle): v0.9.1 — consolidate 6 ready PRs into one CI run (#17194
mrveiss commentedon Sep 22, 2026
Root cause found, and it explains the local/CI split exactly
The mechanism is in
reach_declarations_test._import_every_guard():It imports only modules under
repo_tests/. Butdeclare(...)populates a sharedREGISTRYas an import side effect, and at least one declaration lives elsewhere —prompt-injection-detector-strict-modeis declared inautobot-backend/security/prompt_injection_detector_strict_mode_test.py, which that sweep never reaches.So that declaration enters
REGISTRYonly when something else in the same pytest session has already imported its module. True in CI's whole-suite run. False whenrepo_tests/runs alone.Measured today:
Add the backend module to the same invocation and it collects 36. That is your 224-vs-34 difference in one line: the parametrisation is not over a fixed set, it is over whatever the session happened to import.
Which means the local pass was not merely weaker evidence — it was evidence about a different and smaller set of floors, reported in a form indistinguishable from the full one. A declaration that is never reached produces no
IMPORT_FAILURESentry either, because it is absent rather than broken, and absence is what this file cannot see.MIN_DECLARATIONS = 4is the existing vacuity floor and cannot notice 35 against 36.Filed the fix as #17298 with acceptance criteria — discover declarations wherever they live (or enforce that they all live somewhere the sweep reaches), pin the two counts against each other, and raise the vacuity floor to something that would actually notice one missing. Leaving this issue open as the report; #17298 is the remedy.
Hit it again today on #17187: the floor passed locally at the moment I re-pinned it and failed in CI on the same commit, which is the exact sequence this issue describes. I took the CI number and a direct
git ls-filescount as two independent readings rather than trusting the local run.Refs #17298
fix(ci): put the pricing registry where the guard requires, re-pin tw…
fix(pricing): migrate LLM cost tracking off hardcoded price tables (#…
fix(update): stop the venv provenance marker outliving its install, a…
test(api): no route on /api/long-running can be added without an auth…
fix(deploy): a generated manifest must name paths the node has, not t…
fix(update): stop the venv provenance marker outliving its install, a…
fix(guards): re-pin two reach floors from BOTH their bounds, on this …
fix(deploy): a generated manifest must name paths the node has, not t…
5 remaining items
fix(security): make the live-event admin bypass read the payload auth…
fix(security): scope workflow events to their own channel instead of …
fix(security): a token minted for a purpose is not a login (#17049)
fix(tests): reconcile 11 CI failures with the two gates, one by fixtu…
fix(security): the JWT refusal names which predicate fired (#17049)
docs(threat-model): compress the desktop section, then raise a ceilin…
test(api): restore a third VNC test's subject, and cover the case the…
fix(security): the desktop socket requires desktop-control, not merel…
fix(security): a token minted for a purpose is not a login (#17049)
fix(tests): reconcile 11 CI failures with the two gates, one by fixtu…
fix(security): the JWT refusal names which predicate fired (#17049)
docs(threat-model): compress the desktop section, then raise a ceilin…
test(api): restore a third VNC test's subject, and cover the case the…
fix(security): the desktop socket needs desktop-control, and a purpos…