SDLC gap: config/env changes bypass the cross-model gate, and "fixed" is allowed to mean "edited"

[BaseInfinity]

Incident (2026-08-08)

Autocompact was firing at ~115k on a 1M-context model, repeatedly costing the maintainer working sessions. Root cause: CLAUDE_CODE_AUTO_COMPACT_WINDOW=350000 and CLAUDE_AUTOCOMPACT_PCT_OVERRIDE=35 MULTIPLY.

The fix — deleting the WINDOW key from ~/.claude/settings.json — was correct. It was then reported as fixed while the bug was still live, because an env var cannot be unset in a running process. The user hit the same bug again and reasonably read it as a regression.

Two distinct process defects, both generalizable beyond autocompact.

Defect 1 — "fixed" was allowed to mean "edited"

The verification performed was "the key is gone from settings.json". The verification that mattered was:

env | grep -i compact

Three seconds. It was only run after the user came back and asked.

Checking the config file and claiming it proves the process state is the vacuous-guard defect class — a check that appears to verify X and actually verifies Y. That is precisely what #513 exists to catch in this repo's tests. It shipped inside the workflow that was building the detector.

Proposed rule (/sdlc, Test Failure Recovery or a new short section): before reporting a fix, name the observable that would differ if it were NOT fixed, and go look at that observable. Edited a settings file → read the live process env. Changed a hook → fire it. Changed a threshold → measure it against the real files rather than restating the number. If the only evidence is "I made the edit", the state is submitted, not fixed.

This is cheap, universal, and consumer-relevant — it is not autocompact-specific.

Defect 2 — config/env changes are a review-invisible class

The maintainer asked why neither Fable nor Codex caught it. Neither was ever asked. The change lived in ~/.claude/settings.json: outside the repo, so no diff, no handoff.json, no PR, no review surface at all. Both reviewers only ever see what they are handed.

The knowledge was nevertheless already on disk in this repo:

• .reviews/codex-autocompact-verdict.md (PR docs(ai-setup-lanes): Opus 5 becomes Setup A default, Sonnet 5 moves to Setup B #468, 2026-07-24) quotes the official docs stating that PCT_OVERRIDE only applies proactively when CLAUDE_CODE_AUTO_COMPACT_WINDOW is set — i.e. the multiplication, already reviewed and recorded.
• The maintainer's private memory recorded the live-process-unset problem verbatim, including claude --resume as the real fix.

So this was not a reviewer capability gap. It was a failure to consult an existing review artifact before making an out-of-repo change that no gate covers.

Proposed rule: when a change lands outside the repo (global settings.json, env vars, shell rc, launchctl), it receives no automatic gate. Therefore: read the existing .reviews/ artifacts and memory on that subject BEFORE editing, and state the verification command in the same breath as the change.

Scope note

skills/sdlc/SKILL.md is at 19,356 of its 20,000-byte ceiling, so any addition needs an equal-or-greater trim in the same edit. It is also on the merge denylist, so this cannot ride along on #513 and needs its own PR.

Cross-refs

• The wizard doc embeds a 54,833-byte stale copy of a 19,137-byte skill #513 — the PR whose entire subject is the vacuous-guard defect class
• Autocompact window guidance is wrong for 1M-context models (3 harness layers) #520 — autocompact settings layering (this repo's .claude/settings.json and what the wizard ships)
• docs(ai-setup-lanes): Opus 5 becomes Setup A default, Sonnet 5 moves to Setup B #468 — the earlier Codex autocompact verdict that already contained the answer

[BaseInfinity]

Fable planner leg — spec for #525. Implement verbatim.

Decision

Both defects ship, merged into ONE new section. They are one principle ("a fix is verified by observing the affected observable") with an out-of-repo corollary (Defect 2 is the case where no gate observes it for you). Two sections would duplicate the spine.

Own section, not absorbed. Each candidate home loses: Test Failure Recovery is about failing tests — the incident had no test. Confidence Check is pre-approach, not post-fix. The "CI shepherd's post-deploy verification" doesn't exist as named in the skill — post-deploy verification lives in Deployment Tasks and is deploy-scoped; this rule covers changes with no test and no deploy.

Stale fact confirmed: the 20,000-byte ceiling was deleted by #489 (tests/test-audit-session-load.sh:146 carries the deletion comment; SKILL.md is already 21,808 bytes). The trim constraint is gone. Keep the addition small anyway: the skill loads into every session, the audit still raises TRIM, and bytes are attention.

Wording (verbatim, ready to paste)

## Fixed Means Observed

Before reporting a fix, name the observable that would differ if it were NOT fixed — then go look at it. Settings file edited → read the live process env, not the file. Hook changed → fire it. Threshold changed → measure it against the real files. If the only evidence is "I made the edit", the state is *submitted*, not fixed.

**Out-of-repo changes get no gate** (global `settings.json`, env vars, shell rc, scheduler entries): no diff, no PR, no reviewer ever sees them. Before editing, check `.reviews/` artifacts and memory for prior findings on the subject; state the verification command in the same message as the change; if a live process won't pick up the edit (env vars need a restart), say so instead of "fixed". **Evidence:** 2026-08-08 (#525) — a settings fix was reported fixed while the bug was still live; the answer was already in `.reviews/` from PR #468.

Landing sites

1. skills/sdlc/SKILL.md — insert the section verbatim after ## Debugging Workflow (Systematic) (currently line ~188), before ## Release Planning.
2. cowork/skills/sdlc/SKILL.md — byte-identical copy (existing tests/test-cowork-drift.sh enforces).
3. CLAUDE_CODE_SDLC_WIZARD.md — same text as a ### subsection under ## Testing and Debugging Practices, after ### Debugging Workflow (Systematic Investigation) (~line 2766), before ## Step 8.

No surface states a contradicting rule — pure addition, nothing rewritten. Distinctions to preserve, not merge into: wizard Rule 3 ("run the test", :1005) covers changes that HAVE a test; "Post-Deploy Verification" (:2911) is ARCHITECTURE.md template content and deploy-scoped; "Autocompact Tuning" (~:1052) already documents the WINDOW×PCT multiplication and belongs to #520 — don't touch it.

Considered exclusion: Post-Mortem's formula (… → Test That Proves the Rule → Ship, SKILL.md:239) is contradicted by shipping an untested prose rule. The three-way doc-change test supersedes that line for prose; rewriting it belongs to the issue that owns the three-way test, not #525. Leave it.

Tests

DON'T TEST IT. This is prose whose correctness is a judgement call — bucket three of the settled three-way test, explicitly authorized by the maintainer. A grep could prove presence, never polarity (#539 lesson), and regex guards for prose are retired (#557). Cross-model review is the guard. The only mechanical contract (cowork byte-parity) is already enforced by tests/test-cowork-drift.sh — no new test.

Scope card (#538)

• One issue: SDLC gap: config/env changes bypass the cross-model gate, and "fixed" is allowed to mean "edited" #525 — ship "Fixed Means Observed" + the out-of-repo no-gate corollary into /sdlc.
• Acceptance: the section above present verbatim in skills/sdlc/SKILL.md; cowork copy byte-identical; wizard mirror present; no other section modified.
• Allowed paths: skills/sdlc/SKILL.md, cowork/skills/sdlc/SKILL.md, CLAUDE_CODE_SDLC_WIZARD.md — exactly three files.
• Exclusions: no tests, no hooks, no regex guards (Ship the three /sdlc rules that stop a #539-shaped review loop before round 1 #557); no edits to Define the review contract: severity scale, what blocks, an architecture verdict, and feature creep is always an issue #558 (severity/blocking), Ship the driver/planner authority chain: who outranks whom, and how a model split breaks without a human #559 (authority chain), or Autocompact window guidance is wrong for 1M-context models (3 harness layers) #520 (autocompact layering) surfaces.
• Risk tier: low — additive prose; no gate semantics change (contrast Define the review contract: severity scale, what blocks, an architecture verdict, and feature creep is always an issue #558's medium).
• Estimated diff: ~10 lines × 3 files ≈ 30 lines.

Cut from #525 as written

• The Scope note's trim constraint — obsolete (skills/sdlc/SKILL.md is at its 20,000-byte ceiling — every edit now starts by degrading something else #489 deleted the ceiling).
• "Test Failure Recovery" as the landing site — wrong home; the defect class is fixes without tests.
• Two separate rules — collapsed to one section; the incident narrative stays in the issue, not the skill.
• Kept: "needs its own PR" (merge denylist) still holds.