Skip to content

test(ratchet): nothing fails when a file-size ceiling is raised — only new entries are guarded #16779

Open

Description

@mrveiss

Problem

repo_tests/python_file_size_ratchet_test.py:17 states the rule: "the list may not gain an entry, and no ceiling may be raised (the obvious one)". Only the first half is enforced.

MAX_KNOWN_LARGE_ENTRIES = 497 (:95) is a cardinality guard — it denies a new entry appearing in KNOWN_LARGE and RATCHET_BASELINE together, the two-sided addition described at :78-85. Nothing checks an existing ceiling against its previous value.

test_recorded_ceilings_match_the_files_today (:142) asserts audit_ceilings() returns no problems — the ceilings must match the files as committed. So the passing condition for a file that grew is to edit its baseline to the new size. Growth plus a matching baseline edit is green; the guard cannot tell it from a legitimate re-measure.

Observed on PR #16778, where the rebase raised two ceilings:

-    "autobot-backend/api/knowledge_maintenance.py": 1979,
+    "autobot-backend/api/knowledge_maintenance.py": 1980,
-    "autobot-backend/api/knowledge_search_aggregator.py": 951,
+    "autobot-backend/api/knowledge_search_aggregator.py": 979,

CI does not object. The rule held only because a reviewer read the baseline diff — which is exactly the failure mode :78-85 was written about, one level up: at two raised entries it is visible in a diff, at fifty it is not.

Acceptance criteria

  • A test compares each entry's ceiling against the same entry's value in the merge base and fails when any ceiling is higher.
  • The comparison reads the base from git (the file's committed state), not from a second checked-in copy that would itself need a guard.
  • A deliberate raise is possible but must be explicit and reviewable — an allowlist entry naming the file and the issue that sanctioned it, not a silent number change.
  • Lowering a ceiling stays free and needs no ceremony.
  • The test fails on a synthetic raise, proving it looks — an always-passing version of this guard is the defect it is meant to prevent.
  • The duplicated baseline in scripts/python_file_size_known_large.py is covered by the same check, or the duplication is removed.

Related

The same class of gap as #16354 and #15459: a guard whose documented rule is stronger than its assertion. See docs/developer/RATCHET_BASELINES.md and docs/developer/MEASUREMENT_DISCIPLINE.md.

Activity

  1. mrveiss commented on Sep 16, 2026

    @mrveiss
    OwnerAuthor

    Second observed case, found independently while reviewing #16444: it raises two file-size ratchet ceilings (autobot-backend/api/schemas_system.py 4306→4337, autobot-backend/api/secrets.py 1048→1157) with green CI, exactly the gap this issue describes — test_recorded_ceilings_match_the_files_today only checks the ceiling matches the file as committed, so a growth-plus-matching-baseline-edit passes. Posted the writeup and a blocked label directly on #16444; not asking for anything here beyond the evidence.

  2. added this to the v0.9.5 - CI and guards milestone on Oct 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions