Skip to content

umbrella(guards): 85 enumerating files outside repo_tests/ are covered by no reach check — 5 verified instances #17400

Description

@mrveiss

Parent: #15826.

#15826's meta-guard was built — repo_tests/guard_reach_meta_test.py fails when a new tree-scanning guard arrives with no floor, carries a shrink-only GRANDFATHERED list mirrored against _GRANDFATHERED_BASELINE, and proves itself with a planted unfloored guard. That half is done and is not what this issue is about.

What this issue is about is the boundary that guard states about itself, in its own docstring:

  • Only repo_tests/. 301 tracked .py files repo-wide carry an enumerator — 13 in scripts/, 32 in tools/, 8 in pipeline-scripts/, 32 under autobot-infrastructure/. Those are out of scope here and are not covered by any equivalent check.

That is a correctly declared gap, not a hidden one — and it is still a gap. 85 enumerating files across four directories are covered by no reach check at all, and those directories hold the guards that actually run in pre-commit and CI.

Verified instances

Five guards outside repo_tests/ were read in full and confirmed to render "I could not look" as "I looked and it is clean". Each is a child of this issue.

Guard How a non-measurement becomes a green
scripts/check_websockets_cap.py a parse failure is laundered into "dependency absent", which retires the gate
pipeline-scripts/check_no_literal_ttl_seconds.py a reach counter written for this exact failure mode is never called by main()
scripts/check_nosec_format.py no git return-code check, no floor, and zero output on success
scripts/check_constraint_drift.py rglob anchored to the process CWD, and the scan set is never validated
scripts/audit_api_wiring.py bucket counts never reconcile to the population; no floor on the frontend walk

These are instances, not the disease. The disease is that nothing would have caught any of them, because no check examines that population.

What to build

  • Extend the meta-guard's population beyond repo_tests/ to scripts/, tools/ and pipeline-scripts/, grandfathering the existing unfloored guards in a shrink-only list that fails in both directions, exactly as GRANDFATHERED/_GRANDFATHERED_BASELINE already do.
  • Decide autobot-infrastructure/ (32 files) explicitly — in scope with its own wave, or excluded with the reason recorded in the guard, never left ambiguous.
  • Prove the extension against a planted unfloored guard in each newly covered directory, the way planted_unfloored_guard_test.py already proves the repo_tests/ half. A meta-guard that cannot fail on the thing it exists for is the defect it is checking for.
  • Route the hand-rolled enumerators through tools/lint/_scan_helpers.py:tracked_paths, which already raises EmptyEnumeration rather than returning []. Known call sites: scripts/check_nosec_format.py:107, scripts/check_script_exec_bits.py:99, scripts/check_python_file_size.py:176, scripts/check_shell_file_size.py:145. This is the cheapest systemic fix — it converts each empty-scan hole into a raise for free.
  • Each child below closed with its own evidence.

Relationship to #14896

#14896 proposed tracked_paths for the 35-checker harness and measured "33 of 35 checkers treat an empty scan as a clean scan" on 2026-08-22. tracked_paths now exists (tools/lint/_scan_helpers.py:200-204); the migration of call sites did not follow. #14896 owns the harness consolidation; this issue owns making coverage non-optional and the five verified instances. They should be sequenced, not merged — but they touch overlapping files, so one agent should hold both.

Acceptance criteria

  • A new tree-scanning guard added under scripts/, tools/ or pipeline-scripts/ with no reach floor fails CI.
  • The grandfather list for the newly covered directories fails in both directions — an unrecorded shrink is as much a defect as growth.
  • The extension is proved by a planted unfloored guard per newly covered directory.
  • All five child issues are closed with code evidence.
  • The guard states what it still cannot see, rather than implying full coverage.

Task tree

Sequencing: #17403 and #17404 share the tracked_paths migration and should ride one PR. #17401 and #17402 are independent and small. #17405 cannot start until #17368 lifts the edit interlock. #17403 must be batched with #13554 (same file, different defect).

Related, not duplicated

Provenance

Found by an external-source comparison whose own conclusion was that our doctrine and our remedy plan are both ahead of the reference work; the useful output was this audit of our own tree, not anything adopted. Analysis: docs/research/evidence-first-readiness-gating-toolkit.md.

Numbers quoted from guard_reach_meta_test.py's docstring and read in the tree at bcd7c8dbea; the 79 of 133 in #15826 was not re-measured in this pass.

Activity

  1. mrveiss commented on Sep 24, 2026

    @mrveiss
    OwnerAuthor

    Decision needed: what to do with autobot-infrastructure/ (32 enumerating files)

    This is the checklist item left open deliberately. I have now measured the population rather than
    guessing at it, and the measurement says the directory should not be decided wholesale — it
    holds three different kinds of file and only one of them is what this umbrella is about.

    What is actually in there

    git ls-files 'autobot-infrastructure/*.py' filtered to files containing an enumerator
    (tracked_paths, git ls-files, rglob(, os.walk(, .iterdir() gives 32, matching the
    figure in guard_reach_meta_test.py's docstring. Splitting them by whether anything reads their
    result as a verdict:

    Kind Count Evidence
    Tests of live pre-commit hooks 2 shared/scripts/hooks/pre-commit-no-tracked-symlink_test.py, …-no-gitignore-shadow_test.py. The hooks themselves are wired at .pre-commit-config.yaml:679,692, and the suites are collected by all three CI invocations — ci.yml:404, coverage.yml:107, test-durations.yml:101 — which name autobot-infrastructure/shared/scripts/hooks explicitly since #14884.
    CI-gating 1 shared/scripts/phase_validation_system.py, referenced by .github/workflows/phase_validation.yml.
    One-shot utilities 29 migrations (migrate_config_imports.py, migrate_consolidated_config.py), fixers (fix_bare_excepts.py, fix_whitespace.py), profilers (memory_profiler.py, comprehensive_code_profiler.py), analysis (analyze_duplicates.py, quick_microservice_analysis.py) and similar. None referenced by .pre-commit-config.yaml or any workflow.

    Why the split matters

    The reach floor exists because a guard's green is read as a verdict about the tree. That
    argument applies to the first two rows and does not apply to the third: nobody reads
    analyze_duplicates.py exiting 0 as "the tree is clean". Putting a floor on a migration script
    buys nothing and creates 29 constants that decay — the exact failure #15928 and #17356 are about.

    Options

    A — All 32 in scope, own wave. Consistent, no judgement call at the boundary. Cost: 29 floors
    on scripts whose output is nobody's verdict, each needing re-measurement as the tree moves.

    B — Exclude the whole directory, reason recorded in the guard. Cheapest. Cost: leaves the 2
    hook-test suites and the 1 CI-gating script uncovered, and those are genuinely guard-shaped —
    they are the reason #14884 was filed.

    C — Split (recommended). Bring autobot-infrastructure/shared/scripts/hooks/ into scope in
    the same wave as scripts/ — already CI-collected, already guard-shaped, 2 files. Handle
    phase_validation_system.py as its own entry. Exclude the remaining 29 with the reason recorded
    in the guard as a named decision ("one-shot utilities; exit status is not a verdict about the
    tree"
    ), so the exclusion is auditable rather than silent — the same discipline
    _EXCLUDED_META_FLOORS already applies in reach_floor_migration_test.py.

    Recommendation: C

    It is the only option whose boundary is drawn on "is this output read as a verdict", which is
    the property this umbrella is actually about, rather than on directory name. It also costs the
    least real work — 3 files in scope, not 32.

    Stated limits

    • I did not read the 29 utilities individually; they were classified by name plus the absence
      of any pre-commit/workflow reference. If any of them is invoked from a script or a cron this
      sweep did not search, that reclassifies it and the split should be redrawn.
    • I checked references in .pre-commit-config.yaml and .github/workflows/ only.
    • I initially believed the 2 hook suites ran nowhere. That was wrong — bug(ci): 8 pre-commit hook tests cannot run — the harness invokes a shell builtin through subprocess #14884 found and fixed
      exactly that, and the wiring is present today at the three call sites cited above. Recording the
      correction because the wrong version of this table would have argued for option B.

    cc @mrveiss — this is the one item on #17400 that should not be defaulted silently.

  2. added
    needs-decisionBlocked on an owner decision; options and a recommendation are on the issue
    on Sep 24, 2026
  3. added this to the v0.9.5 - CI and guards milestone on Oct 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions