Audit the test suite itself — 33,296 lines of bash that has certified its own defects at least six times

[BaseInfinity]

Requested 2026-08-07. Note: a CI/CD audit already exists as #491 — this is
deliberately the other half. #491 asks whether the pipeline can see defects; this
asks whether the tests can, independent of how they are invoked.

Scale

Suites	65
Total lines of bash	33,296
Largest	test-hooks.sh 3,359 · test-doc-consistency.sh 3,036 · test-workflow-triggers.sh 2,510 · test-self-update.sh 2,306

There is no unit/integration layering here — it is 33k lines of assertions guarding a
repo with no application code. That is a large enough surface to deserve its own
audit, and it has never had one.

Why now — six recorded cases where the suite certified a defect

Every one of these is from this repo's own history, not hypothetical:

1. Two "0 failing" reports from suites that ran zero tests. timeout(1) does not
   exist on macOS, so the runner exited before executing anything and reported success.
2. Assertions inert on Linux, green on macOS (Portable-grep audit: \` is a GNU anchor, a BSD literal — assertions can be inert on Linux while green on macOS #488). \` is a literal
   backtick to BSD grep and a start-of-buffer anchor to GNU grep, so a required-token
   check matched nothing in CI while passing locally. The suite reported 104/104.
3. A hand-listed hook roster that silently stopped covering a hook —
   test-hook-stdin-bounded.sh omitted model-effort-check.sh, which is precisely
   why the v1.94.0 stdin-hang regression shipped through the release built to fix it.
4. Three vacuous version-marker guards, each proven hollow by review and pulled
   from v1.94.0 (Version-marker guard: three markers sat at v1.90.0 for three releases, and three attempts at a guard were each proven vacuous #493). One returned success when its extraction step was missing, so
   8 of 9 extractors were unchecked.
5. Three detector regressions in PR fix(hooks,docs): three shipped defects, and five rounds of fixing the tests meant to catch them #497, in three consecutive rounds, each caught
   only by adversarial reviewer probing — never by CI. Two were false positives that
   blocked honest prose.
6. A line-pinned assertion (sed -n '113p') that broke when an unrelated edit
   five lines earlier shifted the file. It asserted a line number, not a property.

The pattern is consistent: the suite is good at checking that text exists and bad at
checking that a guarantee holds. Six times it has reported green over a live defect.

What the audit should actually ask

• Which assertions are vacuous today? Mechanical first pass: mutate the thing each
  test claims to protect and confirm the test goes red. Any test that cannot be made to
  fail is not a test. This is the single highest-value pass and it is scriptable.
• Which assertions are pinned to positions rather than properties? Line numbers,
  fixed offsets, exact counts that drift.
• Which are portability traps? Portable-grep audit: \` is a GNU anchor, a BSD literal — assertions can be inert on Linux while green on macOS #488 was one instance; the audit should sweep for
  the class (BSD vs GNU grep/sed/date/stat, bash 3.2 on macOS, absent timeout).
• Which fixtures can corrupt the real tree? At least one suite mutates cwd when
  its tmpdir creation fails under sandbox — a test that damages the repo it is testing.
• What is untested? 33k lines of assertions is not the same as coverage. Name the
  guarantees this repo makes that nothing checks.
• What should be deleted? Some of this is duplicative (Replace 2,443 lines of hand-rolled doc guards with declarative consistency checking #485 measures six functions
  asserting one fact). A smaller suite that fails honestly beats a larger one that does not.

Suggested method

Run it as a parallel blind dual review (Fable + Codex, neither seeing the other's
findings), which is the protocol that produced findings 3, 5 and 6 above. Feed each a
slice of the suite rather than the whole 33k lines. Prioritise by blast radius: suites
guarding hooks/ and skills/ first, since those ship.

Do not accept "the suite passes" as evidence during this audit. That is the exact
claim under investigation.

Refs #491 (CI/CD half), #488 (portability class), #493 (vacuous guards), #498
(detector fixtures), #485 (duplication).

[BaseInfinity]

Evidence from PR #509's review, 2026-08-08 — two more instances of the class

Both found by cross-model review, neither by CI or by the author.

1. A guard defeated three times, each time by prose ABOUT the rule

test_skill_keeps_exempted_landmines pins the #364/#437 landmines in skills/sdlc/SKILL.md.
Three successively-tightened versions were each defeated by a reviewer:

Version	Assertion	What passed it
v1	any mention of commit_sha	# TODO: add commit_sha
v2	CERTIFIED and commit_sha co-occur	On CERTIFIED mentions commit_sha
v3	commit_sha followed by rev-parse	Discussion reference: commit_sha: git rev-parse HEAD.

The generalisable finding: a guard that names a concept is satisfied by discussion of the
concept. Every version above matched text that talks about the rule instead of stating it.
The working version requires all four elements of the instruction on one line.

Second-order lesson, and the more useful one for this audit: my own mutation testing
declared v2 and v3 "proven" — because I mutated by deleting every occurrence of the token
at once, so the test went red for the wrong reason. A mutation that removes both the rule
and the prose about the rule cannot distinguish a tight assertion from a loose one. Any
audit that re-verifies existing tests must mutate the specific thing the assertion claims
to protect, not the whole concept.

2. A portability defect that only reproduces under a non-default locale

tests/test-community-scanner.sh and tests/test-token-spike.sh capture with 2>&1. Under
an inherited LC_ALL=C.UTF-8, bash emits warning: setlocale: cannot change locale —
C.UTF-8 is glibc-only and absent on macOS — and that warning lands inside the value being
asserted. 14 false failures in one file, 2 in the other.

I could not reproduce it because my locale is en_US.UTF-8. The reviewer could, and pressed
for confirmation rather than accepting "passes here". Fixed by filtering that specific
warning at all 26 capture sites, rather than forcing LC_ALL=C — forcing a locale would
also mask real stderr from the scripts under test.

Same class as #488 (BSD vs GNU grep): an assertion whose result depends on the environment
of whoever runs it. #488 was a locale-adjacent divergence between platforms; this is a
divergence between shells on the same platform. Worth one audit pass covering both.

Suggested additions to this issue's scope

• Mutate the specific claim, not the concept. Re-verifying a test by deleting every
  occurrence of its subject is the methodology that produced two false "proven" results here.
• Run the suite under a hostile environment, not just the auditor's. At minimum
  LC_ALL=C.UTF-8 and LC_ALL=C; ideally also on Linux, since Portable-grep audit: \` is a GNU anchor, a BSD literal — assertions can be inert on Linux while green on macOS #488 was inert on one
  platform and green on the other.
• Check for assertions satisfiable by prose — any test grepping documentation for a
  concept name rather than the concept's operative form.

[BaseInfinity]

Concrete instance found in-session: mktemp template with a suffix is non-portable, and the failure is invisible

tests/test-self-update.sh:85

temp_yaml=$(mktemp "${TMPDIR:-/tmp}/wizard-workflow-XXXXXX.yml")

BSD mktemp (macOS) only substitutes the XXXXXX run when it is at the end of the template. With .yml trailing, it does not substitute — it tries to create the literal path wizard-workflow-XXXXXX.yml. Observed failure, running the suite while a second process happened to be running it too:

mktemp: mkstemp failed on /var/.../T//wizard-workflow-XXXXXX.yml: File exists

Two consequences, the second worse than the first:

1. It is not a flake, it is a guaranteed collision. Any two concurrent runs on macOS race for one fixed filename. In isolation the suite passes 158/158, so this only surfaces when something else is running — which is exactly the situation during a cross-model review, when a reviewer with danger-full-access may run the suite itself.
2. It sits under set -e in a command substitution, so the run dies mid-suite with a bare mktemp: line and no FAIL: marker. A harness counting FAIL: lines would score this as passing. That is the same "the test suite certifies its own defects" shape this issue is about, arriving through the fixture rather than the assertion.

Fix is mktemp "${TMPDIR:-/tmp}/wizard-workflow-XXXXXX" then rename, or mktemp -t. Sweep the whole suite for mktemp templates with a trailing suffix rather than patching this one site — and add a guard, since the local-run signal for this class is silence.

Not fixing now (unrelated to the PR in flight); logged here so the sweep picks it up.

[BaseInfinity]

Closing into #565, on unanimous cross-model ruling.

This issue asks for an audit of 33,296 lines of bash that "has certified its own defects at least six times." The premise is right and the scope is unbounded, which is why it never started. #565 asks the same question with a method attached: run every test through the three-way call from #561, and for each survivor, name the wrong version it catches. If that sentence cannot be written, the test does not survive.

Two things from here carry into #565 rather than dying with it:

• The six-self-certified-defects evidence. That count is now seven — the seventh was found inside PR Scope TDD RED to where a RED mutation is writable, and delete the hook that blocked the maintainer from asking for it #567, in a guard written that night with this exact defect class in mind, caught by a reviewer mutating the file rather than reading the assertion.
• The portability and real-tree-safety requirements. Both are live: Portable-grep audit: \` is a GNU anchor, a BSD literal — assertions can be inert on Linux while green on macOS #488 records ``` as a GNU anchor and a BSD literal, so assertions can be inert on Linux while green on macOS; and test-persist-score-history.sh fails open: its mktemp/cd fallback can run git against the real repo #566 records a fixture that runs its git commands against the real repo when its sandbox creation fails.

The audit is blocked by #561 shipping, which it now has.