Replace 2,443 lines of hand-rolled doc guards with declarative consistency checking

[BaseInfinity]

The problem, measured

tests/test-doc-consistency.sh is 2,443 lines and 170 test functions. A large fraction encode one pattern: "this value must be stated consistently across the shipped files."

Written out by hand, per value, per file. Examples currently in the tree:

• test_ai_setup_lanes_reviewer_is_gpt56
• test_readme_reviewer_is_gpt56
• test_wizard_doc_reviewer_is_gpt56
• test_skill_files_reviewer_is_gpt56
• test_agents_md_reviewer_is_gpt56
• test_claude_md_reviewer_is_gpt56

Six functions, one fact. The same shape repeats for Sonnet 5's default effort, Opus 1M framing, model-pin recommendations, and now driver and reviewer effort.

Why this is worth fixing, with tonight's evidence

Changing one value — the Opus 5 driver effort default from xhigh to high/medium — took three cross-model review rounds and produced nine separate contradictions across shipped files. Every round followed the same script: fix the files I remembered, add or update an anchor guard, declare it done, reviewer finds more.

The reviewer's diagnosis was exact: "The guard passes because it does not cover this file."

That is not carelessness that better attention fixes. A guard enumerating locations cannot detect a location nobody enumerated. The ninth instance was found only after the guard was widened to scan all shipped files for the assertion shape rather than checking a list of anchors — and it found one that two independent reviewers had missed.

Same class of defect as ROADMAP #495 (the reviewer-effort denylist arms race) and #490 (tests asserting text rather than behaviour). Three symptoms, one cause: the check is written per-incident instead of per-property.

Proposed shape

A declarative manifest of shipped facts plus one generic checker. Roughly:

- fact: reviewer_model
  must_say: ["GPT-5.6", "Sol"]
  must_not_say: ["GPT-5.5", "GPT-5.4"]
  scope: shipped            # resolved from `npm pack --dry-run`, not `files`
  except: [historical-citation-markers]

- fact: driver_effort_default
  must_say: ["`high`", "`medium`"]
  must_not_assert: "Opus 5 .{0,40}`xhigh`"
  allow_qualified: ["escalat", "planner", "not the default"]

Adding a new invariant becomes a data change, not thirty lines of bash. Changing a value becomes editing one row and watching every offending file fail at once, rather than discovering them across three review rounds.

Requirements, drawn from what actually bit us

1. Scope must be derived, not listed. npm pack --dry-run, not package.json's files — README ships despite being absent from files, and that omission directly caused a missed contradiction tonight.
2. Must distinguish assertion from mention. Documenting a change necessarily mentions the old value. xhigh appearing is fine; "Opus 5 defaults to xhigh" is not. Historical citations (the GPT-5.4 benchmark audit) must survive untouched — there are already guards specifically protecting them.
3. Must fail loudly on unknown constructs rather than silently skipping, per the lesson in AGENTS.md Code Review Rule 2.
4. Must be mutation-tested. Reintroduce each invariant's violation and confirm it fails. Non-negotiable given this repo's history of vacuous guards.

Explicitly NOT proposed

Deleting the existing 170 tests wholesale. Many encode genuinely bespoke logic — byte ceilings, drift between skill copies, workflow structure — that does not fit this pattern. Migrate only the value-consistency subset, and count it first rather than trusting this issue's estimate.

Related

• ROADMAP --user-approved: let the human say the thing HARD_DENY demands #495 (denylist → positive anchors; same arms race, narrower)
• ROADMAP Ship a model-ADOPTION policy, not just a model recommendation — "if it works, don't break it" #490 (testing contract; tests asserting text not behaviour)
• Make GitHub Issues the source of truth; ROADMAP.md becomes a view, not a store #482 (GH issues as source of truth)
• AGENTS.md → Code Review Rules 2 and 3

[BaseInfinity]

Measured: a deterministic rule catches most of this instantly

Maintainer's framing, and it is the right one: "we shouldn't use AI to do all these regression tests UNLESS we cannot do it any other way, but I highly doubt that."

Tested rather than assumed. Extracted the eight shipped files at their pre-fix state (c3c9d02) and ran the single widened grep rule against them:

README.md: 1
AI_SETUP_LANES.md: 1
CLAUDE_CODE_SDLC_WIZARD.md: 1
skills/sdlc/SKILL.md: 1
skills/update/SKILL.md: 1
TOTAL caught in one run: 5

Five contradictions, one rule, milliseconds. Those same five cost three cross-model review rounds — roughly 21 minutes of model time plus tokens — discovered one batch at a time, each round ending with me wrongly declaring the sweep complete.

The remainder are not harder, just differently shaped: "effortLevel": "xhigh" inside JSON settings templates needs a value rule rather than a prose-adjacency rule. Still deterministic. Still milliseconds.

Where the line actually falls

Worth stating precisely, because "no AI" would be the wrong conclusion. Across the same session, cross-model review caught three things no linter could have:

• A changelog label rewritten from [1.90.0] to [1.91.0] while still describing 1.90's content. Syntactically fine; semantically false. Needs a reader who knows what those versions shipped.
• README.md ships via npm despite absence from package.json's files — verified with npm pack --dry-run. That is a fact about tooling behaviour, not a pattern in a file.
• A guard passing all 15 of its fixture assertions while reading nothing from the document it guarded. Required executing it and reasoning about why green was wrong.

The split, then:

Class	Tool	Why
A value must be stated consistently across shipped files	Deterministic rule	Enumerable, mechanical, instant, zero marginal cost
Is this claim true / is this test vacuous / does this ship	Cross-model review	Requires judgement, execution, or external verification

Tonight inverted that: AI did the mechanical work three times over, while the judgement calls were the minority of its findings. This issue exists to move the first row off the reviewer's plate entirely — not to reduce review, but to stop spending it on grep.

Consequence for prioritisation

This raises #485 above the other open research items. Every future value change — a model rename, an effort default, a version pin — pays the same three-round tax until it is fixed. That tax is now measured, not asserted.

[BaseInfinity]

Evidence from #520 (2026-08-09) — a concrete failure mode for the hand-rolled guards, and the pattern that terminated it.

Three doc-consistency assertions written for #520 were defeated three separate times by cross-model review, each time by a different attack against the same design:

1. Vocabulary collision — a whole-file disabl grep was satisfied by the unrelated disableAllHooks text, so the guarded sentence could be deleted with the assertion green.
2. Deletion — the guards were denylists ("reject a bad value"), so removing the guidance entirely passed at 129/129.
3. Semantic inversion — after re-anchoring to ^1\..*200000[^.]*disabl, the reviewer replaced the claim with "a window under 200000 does not disable autocompact". Every token still co-occurs on the right list item. Still 129/129 green.

(3) is the interesting one for this issue: token proximity cannot carry polarity. A negation denylist would have been a fourth patch in the same arms race and loses to "fails to disable" or a clause reorder.

What actually terminated it: pin the load-bearing sentence verbatim — whitespace-normalized grep -F against a constant defined next to the assertion — layered over the structural anchors. Structural anchors catch deletion; the literal pin catches inversion and rewording. The only string that satisfies a literal match for the true claim is the true claim.

Declared residual, which is the part this issue should solve: a textual guard cannot detect a contradicting sentence added elsewhere in the document. The pinned sentence still matches. That is a genuine gap in the grep-based approach, not a gap in this particular guard, and it's the strongest argument yet for declarative consistency checking over 2,443 lines of hand-rolled bash.

Nineteen mutations now cover this surface, including all three attacks above. Full trail in .reviews/codex-520-r*.md.