Skip to content

bug(guards): reach_declarations_test passes locally on a commit CI fails — a local pass is not evidence #17144

Description

@mrveiss

Problem

On #17072 head ff296b7142, CI fails:

ReachFloorError: [prompt-injection-detector-strict-mode] floor 2940 sits 304 below
its live population of 3244 tracked backend python files (tests excluded),
which exceeds the declared allowance of 300 (skips=0 + growth=300).
FAILED repo_tests/reach_declarations_test.py::test_every_declared_floor_is_pinned_to_its_population[prompt-injection-detector-strict-mode]

Running that same test against that same commit in a detached worktree passes — twice, in two
separate worktrees (224 passed, then 34 passed for the parametrized floor test alone).

The population data is present locally. In the same worktree, the guard's own discovery command
returns the same number CI reports:

git ls-files -- 'autobot-backend/*.py' | grep -v '_test\.py$' | wc -l   ->  3244

So the guard had the failing number available and did not fail on it.

Why this matters

A guard that passes locally and fails in CI on identical input is worse than one that simply fails:
it converts local verification into false assurance. Working from that local pass, the obvious
conclusion was "the floor is fine, something else in shard 12 is failing" — which is wrong, and
would have sent someone looking for a second, non-existent defect while the real one sat in plain
sight in the error message.

This also undermines the pre-push hook's value for this class of guard: a developer who runs these
tests before pushing gets a green that does not predict CI.

What is NOT the cause

  • Main moving under the branch: git log 0f5b45cab..origin/main is empty and behind-ok reports
    behind=0.
  • A stale checkout: the worktree was created fresh from origin/issue-17043-17056-approval-consolidation
    and git log --oneline -1 confirmed ff296b7142.
  • The discovery command failing outright: it returns 3244 in that worktree.

Unresolved

I did not determine why the guard does not act on the number its own discovery returns. Candidates
worth checking, none confirmed: repo_root()'s lru_cache interacting with worktrees;
reach_declarations_test driving discover against a substitute root (its docstring mentions
being driven against an empty directory for #15826) so the real root is never measured; or
scrubbed_git_env() behaving differently under a worktree than under a plain checkout.

Stating that as unresolved rather than picking the most plausible one — a guessed cause here would
send the fix at the wrong mechanism.

Acceptance criteria

  • The reach floor guard produces the same verdict locally and in CI for the same commit,
    demonstrated on ff296b7142 (or a reconstruction of it) failing in both.
  • If the divergence is worktree-specific, the guard either works correctly in a worktree or
    refuses to run there with an explicit message — never a silent pass.
  • A test asserts the guard fails when the population exceeds floor+growth, driven through the
    same entry point CI uses, so a future vacuous pass is caught.

Notes

Found landing #17072. Related: #17142 (the same floors re-pinned three times in one night) and
#15928 (floors set below their own discovery).

Activity

  1. mrveiss commented on Sep 22, 2026

    @mrveiss
    OwnerAuthor

    Root cause found, and it explains the local/CI split exactly

    The mechanism is in reach_declarations_test._import_every_guard():

    for module in pkgutil.iter_modules([str(_REPO_TESTS)]):

    It imports only modules under repo_tests/. But declare(...) populates a shared REGISTRY as an import side effect, and at least one declaration lives elsewhere — prompt-injection-detector-strict-mode is declared in autobot-backend/security/prompt_injection_detector_strict_mode_test.py, which that sweep never reaches.

    So that declaration enters REGISTRY only when something else in the same pytest session has already imported its module. True in CI's whole-suite run. False when repo_tests/ runs alone.

    Measured today:

    $ pytest repo_tests/reach_declarations_test.py::test_every_declared_floor_is_pinned_to_its_population -v | grep -c PASSED
    35
    $ ... | grep -i prompt-inject
    (no output)
    

    Add the backend module to the same invocation and it collects 36. That is your 224-vs-34 difference in one line: the parametrisation is not over a fixed set, it is over whatever the session happened to import.

    Which means the local pass was not merely weaker evidence — it was evidence about a different and smaller set of floors, reported in a form indistinguishable from the full one. A declaration that is never reached produces no IMPORT_FAILURES entry either, because it is absent rather than broken, and absence is what this file cannot see.

    MIN_DECLARATIONS = 4 is the existing vacuity floor and cannot notice 35 against 36.

    Filed the fix as #17298 with acceptance criteria — discover declarations wherever they live (or enforce that they all live somewhere the sweep reaches), pin the two counts against each other, and raise the vacuity floor to something that would actually notice one missing. Leaving this issue open as the report; #17298 is the remedy.

    Hit it again today on #17187: the floor passed locally at the moment I re-pinned it and failed in CI on the same commit, which is the exact sequence this issue describes. I took the CI number and a direct git ls-files count as two independent readings rather than trusting the local run.

    Refs #17298

  2. 5 remaining items

  3. added this to the v0.9.5 - CI and guards milestone on Oct 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    backendbugSomething isn't working

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions