bug(ci): the blocking stale-baseline audit reds ssot-coverage on any PR touching 8% of the repo #14912

Activity

1. mrveiss

   added

   bugSomething isn't workingSomething isn't working

   on Aug 23, 2026
2. github-actions

   added

   infrastructure

   on Aug 23, 2026
3. mrveiss commented on Aug 23, 2026

   

   mrveiss

   on Aug 23, 2026

   OwnerAuthor

   More actions

   Body rewritten with a corrected account. Two changes of substance:

   1. The guard did not cause the base red. It fired on real drift and reported accurately on the PR that caused it; the base red came from a merge issued against an already-failing head. The first version of this issue misattributed that, and the record should not stand wrong. The calibration finding — recovery cost — stands on its own.
   2. Checked the 'should the audit run on the causing PR' question rather than assuming it. It already does: ssot-coverage ran on fix(provision): route every component removal through one data-guarded task, so an unknown fact cannot delete a database (#14856) #14890's own head and failed there. The real gap is that SSOT Configuration Compliance is not one of the ten required contexts on Dev_new_gui, so the detector, the stale audit and the new no-growth guard are all advisory at the protection layer. Recorded with the ci: ~25 workflow runs per PR head movement saturate GitHub-hosted concurrency and manufacture threshold-test failures #13388 path-filter constraint and the existing required-context-shim precedent; promotion is an owner decision and deliberately out of this issue's scope.
4. mrveiss commented on Aug 23, 2026

   

   mrveiss

   on Aug 23, 2026

   OwnerAuthor

   More actions

   Prototyped hv_pruned_baseline against a read-only copy of the merged library while blocked on #14910. Three properties confirmed empirically, not just by construction:

   property	result
   brand-new key appears in pruned output	0 — cannot add
   any pruned count raised above its baseline	none — cannot raise
   key seen 3x but baselined 2	pruned to 2, not 3

   The implementation iterates only keys already in HV_BASELINE (so it can never introduce one) and emits min(baseline_count, seen_count) (so it can never raise one). hv_partition increments HV_BASELINE_SEEN before the allowance test, so seen counts every finding for a key — which is what makes min correct rather than accidentally lossy.

   Safety gap the prototype exposed, now part of the design. Prune drops any baselined key the scan did not see. That is the intended behaviour against a full tree scan, but it means a broken or partial scan silently guts the baseline: in the prototype a deliberately tiny simulated scan reduced 1411 entries to 2. A scan that fails to load rules, or whose scan directories are missing, produces few or zero findings and would take the whole record with it — and the no-growth guard would not object, because shrinking is allowed by design.

   Two additional acceptance criteria:

   • Prune REFUSES on an empty scan. A tree carrying 1411 baselined findings cannot legitimately drop to zero in one change, and an empty result is indistinguishable from a broken detector — the exact failure shape these guards exist to catch.
   • Prune reports how many entries it will remove before writing, so removing 1 entry and removing 1400 are visibly different in the diff and in review.

   Still holding on #14910 as instructed — it touches the same baseline file. Waiter armed on its merge.
5. mrveiss commented on Aug 23, 2026

   

   mrveiss

   on Aug 23, 2026

   OwnerAuthor

   More actions

   Implemented in PR #14915 (26c7064a0), branched fresh off origin/Dev_new_gui (192a2d12c) after #14910 merged.

   All acceptance criteria met, each mutation-proved rather than asserted:

   criterion	proof
   --prune-baseline removal-only	real unbaselined violation in tree -> prune -> key not added, still reported as a violation, no-growth green
   cannot raise a count	key baselined 1, found twice -> stays 1 (min(baseline, seen))
   recovery loop	strand -> audit RED -> prune -> audit GREEN, baseline byte-identical to pre-mutation
   audit names command + entries	failure output carries both, plus why the command is safe
   refuses on empty scan	hermetic tree with no scan dirs -> refuses, baseline unwritten
   docs	HARDCODING_PREVENTION.md gains the recovery section

   The removal-only property is structural, not a check: hv_pruned_baseline iterates keys already in the baseline (cannot introduce one) and emits min(baseline_count, seen_count) (cannot raise one). That matters because a prune able to add or increment would be the bypass check_baseline_no_growth.sh exists to prevent.

   The empty-scan refusal came out of prototyping, not review: prune drops every key the scan did not see, so a broken detector or a moved scan directory would take all 1410 entries with it — and the no-growth guard would not object, since shrinking is allowed by design.

   The required-context gap recorded in this issue is not addressed by #14915 and remains open here as an owner decision. Separately, #14914 was filed for a distinct finding: 8 of 12 rule classes never block, and 9 unbaselined violations sit on the base right now under "status": "pass".
6. mrveiss

   added 2 commits that reference this issue on Aug 23, 2026

   fix(ci): refuse to scan a partial tree, so a moved directory cannot g…

   ...

   c7de43d

   fix(ci): give the stale-baseline audit a one-command recovery (#14912) (

   ...

   c83c2e3
7. github-actions commented on Aug 23, 2026

   

   github-actionsbot

   on Aug 23, 2026 – with GitHub Actions

   Contributor

   More actions

   PR #14915 (merged to Dev_new_gui) references this issue with a close keyword.

   fix(ci): give the stale-baseline audit a one-command recovery (#14912)

   If this issue is fully resolved, close it manually. If work remains, no action is needed.
8. mrveiss commented on Aug 23, 2026

   

   mrveiss

   on Aug 23, 2026

   OwnerAuthor

   More actions

   Delivered and merged as c83c2e39e (PR #14915).

   • CI: 39 success / 10 skipped / 0 failures, deduped to the latest run per check name on head c7de43dcd. All three touched steps green on the runner: the detector, the stale-entry audit, and the no-growth guard.
   • Review: independent code-reviewer pass. Removal-only confirmed structural, not merely tested — one write path, no early-write error branch, cannot add a key, cannot raise a count, no escape found for either property. Header round-trips byte-for-byte.

   What this fixes: the recovery cost identified here. A fixed violation can now drop its baseline line with one command, and the audit failure names both the offending entries and the exact command.

   A HIGH finding was raised in review and fixed before merge, not deferred.

   The empty-scan refusal only caught a scan returning literally zero. hv_scan_tree treats a missing root as a silent no-op, so one moved SCAN_DIRS entry yields nothing while the other five yield hundreds — the total stays non-zero, the refusal never fires, and min(kept, 0) drops every key beneath the unreached directory. Reproduced hermetically before fixing:

   hardcoded-values: baseline pruned — 5 key(s) -> 2 key(s), 3 removed
     exit rc=0
   

   --audit-baseline then passed, because every survivor still matched, and the no-growth guard stayed silent because shrinking is allowed by design. Nothing between them checked magnitude. Real blast radius on the committed baseline: 554 entries under autobot-backend, 405 under autobot-slm-backend, 337 under autobot-infrastructure.

   The fix guards every mode, not just prune — a deliberate widening of the suggested fix, and the right one. A partial scan corrupts all three: --json under-reports the counts ssot-coverage.yml gates on, and --audit-baseline reports the unreached directory's entries as STALE. That last one compounds: the audit falsely accuses real entries, and its own remedy is to prune, which would then delete them. Two mechanisms that each look correct in isolation.

   Landed message, verified in base:

   FATAL: scan directory '<dir>' does not exist under <root>.
     Refusing to scan a partial tree: findings under it would be missing,
     its baseline entries would look stale, and a prune would delete them.
   

   A proportional-loss guard was considered and declined, with the reasoning recorded in the code: no percentage threshold has a non-arbitrary value, it would block the one legitimate large prune (a sweep fixing many violations at once), and any loss under the threshold still passes. The existence check removes the cause rather than rationing the symptom.

   Also corrected: the guard's own justification comment cited "a rules file that failed to load", which is unreachable — the script is already fatal on an unsourceable library before scanning. Naming a risk the code already forecloses makes the remaining real one look handled.

   Tests: 8 added, parametrized over all six scan dirs so the guard cannot silently cover five, plus a SCAN_DIRS drift test. Includes the byte-identical no-op prune the PR body had claimed as a proof point with no committed test behind it. Also folded in: LC_ALL=C on the prune sort (unpinned collation could reorder every line in a different CI image) and the temp file added to the cleanup trap.

   This PR does not modify the baseline itself — 1433 lines before and after.

   Remaining in this area: #14914, where STATUS and the workflow exit are keyed on ssot_violations alone, so 8 of 12 emit classes never block and the base currently reports pass over 9 unbaselined violations.
9. mrveiss

   closed this as completedon Aug 23, 2026