Repository navigation
bug(guards): reach_declarations_test passes locally on a commit CI fails — a local pass is not evidence #17144
Description
Activity
- added 4 commits that reference this issue
on Sep 20, 2026 Root cause found, and it explains the local/CI split exactly
The mechanism is in
reach_declarations_test._import_every_guard():for module in pkgutil.iter_modules([str(_REPO_TESTS)]):
It imports only modules under
repo_tests/. Butdeclare(...)populates a sharedREGISTRYas an import side effect, and at least one declaration lives elsewhere —prompt-injection-detector-strict-modeis declared inautobot-backend/security/prompt_injection_detector_strict_mode_test.py, which that sweep never reaches.So that declaration enters
REGISTRYonly when something else in the same pytest session has already imported its module. True in CI's whole-suite run. False whenrepo_tests/runs alone.Measured today:
$ pytest repo_tests/reach_declarations_test.py::test_every_declared_floor_is_pinned_to_its_population -v | grep -c PASSED 35 $ ... | grep -i prompt-inject (no output)Add the backend module to the same invocation and it collects 36. That is your 224-vs-34 difference in one line: the parametrisation is not over a fixed set, it is over whatever the session happened to import.
Which means the local pass was not merely weaker evidence — it was evidence about a different and smaller set of floors, reported in a form indistinguishable from the full one. A declaration that is never reached produces no
IMPORT_FAILURESentry either, because it is absent rather than broken, and absence is what this file cannot see.MIN_DECLARATIONS = 4is the existing vacuity floor and cannot notice 35 against 36.Filed the fix as #17298 with acceptance criteria — discover declarations wherever they live (or enforce that they all live somewhere the sweep reaches), pin the two counts against each other, and raise the vacuity floor to something that would actually notice one missing. Leaving this issue open as the report; #17298 is the remedy.
Hit it again today on #17187: the floor passed locally at the moment I re-pinned it and failed in CI on the same commit, which is the exact sequence this issue describes. I took the CI number and a direct
git ls-filescount as two independent readings rather than trusting the local run.Refs #17298
- added 8 commits that reference this issue
on Sep 22, 2026 5 remaining items
- added 14 commits that reference this issue
on Sep 24, 2026
Problem
On #17072 head
ff296b7142, CI fails:Running that same test against that same commit in a detached worktree passes — twice, in two
separate worktrees (
224 passed, then34 passedfor the parametrized floor test alone).The population data is present locally. In the same worktree, the guard's own discovery command
returns the same number CI reports:
So the guard had the failing number available and did not fail on it.
Why this matters
A guard that passes locally and fails in CI on identical input is worse than one that simply fails:
it converts local verification into false assurance. Working from that local pass, the obvious
conclusion was "the floor is fine, something else in shard 12 is failing" — which is wrong, and
would have sent someone looking for a second, non-existent defect while the real one sat in plain
sight in the error message.
This also undermines the pre-push hook's value for this class of guard: a developer who runs these
tests before pushing gets a green that does not predict CI.
What is NOT the cause
git log 0f5b45cab..origin/mainis empty andbehind-okreportsbehind=0.origin/issue-17043-17056-approval-consolidationand
git log --oneline -1confirmedff296b7142.Unresolved
I did not determine why the guard does not act on the number its own discovery returns. Candidates
worth checking, none confirmed:
repo_root()'slru_cacheinteracting with worktrees;reach_declarations_testdrivingdiscoveragainst a substitute root (its docstring mentionsbeing driven against an empty directory for #15826) so the real root is never measured; or
scrubbed_git_env()behaving differently under a worktree than under a plain checkout.Stating that as unresolved rather than picking the most plausible one — a guessed cause here would
send the fix at the wrong mechanism.
Acceptance criteria
demonstrated on
ff296b7142(or a reconstruction of it) failing in both.refuses to run there with an explicit message — never a silent pass.
same entry point CI uses, so a future vacuous pass is caught.
Notes
Found landing #17072. Related: #17142 (the same floors re-pinned three times in one night) and
#15928 (floors set below their own discovery).