roadmap(#220-222): Anthropic 2026-04-23 post-mortem learnings

ROADMAP.md

@@ -238,6 +238,9 @@ Living tracker of projects shipped using this wizard. **Rule:** only list projec
 | 90 | Distribution Channels Sprint | Multi-channel expansion beyond npm: (a) awesome-claude-code + awesome-claude-skills + SkillsMP submissions, (b) `install.sh` curl script, (c) Homebrew tap (`homebrew-sdlc-wizard` repo), (d) GitHub CLI extension (`gh-sdlc-wizard` repo, 30min), (e) GitHub Releases with CI automation, (f) Scoop bucket for Windows, (g) AUR package for Arch. Skip: Flatpak (rejects CLI), Snap (sandbox conflicts), .deb/.rpm (curl covers), pip/cargo/go (wrong ecosystem). Meta-tool: GH Actions release workflow auto-updates all channels on npm publish |
 | 91 | Multi-Agent Adapter Layer | Skills (SKILL.md) port directly to any AI agent. Hooks need per-agent adaptation: Codex (notify config), Cursor (.cursorrules + marketplace), Windsurf (.windsurfrules), Aider (conventions). Strategy: adapter layers per agent, not forks. Cursor marketplace submission is medium priority (1-2 days). Codex plugin: Stefan building this weekend. Also consider: Continue.dev, Amazon Q, GitHub Copilot Workspace |
 | 194 | ~~opencode SDLC Adapter~~ SUPERSEDED → see Next Up #9 | Expanded and promoted to Next Up #9 on 2026-04-22 with privacy-first / any-backend framing (local Ollama/LM Studio/vLLM, enterprise Azure OpenAI + internal gateways, hosted OSS). Original scope retained there plus backend-matrix proof phase |
+| 220 | Token-spike anomaly detection (post-mortem 2026-04-23) | Anthropic's 2026-04-23 post-mortem ([anthropic.com/engineering/april-23-postmortem](https://www.anthropic.com/engineering/april-23-postmortem)) showed a caching bug "continuously dropped thinking blocks from subsequent requests" and caused unexpected usage drain — detectable only via token-consumption spikes. Our effectiveness scoreboard (#80) logs score/quality events but does not track token burn. Add a per-session token-delta signal to `catches.jsonl` (or sibling `token-history.jsonl`): record `input_tokens` / `output_tokens` / `cached_read_tokens` / `cached_creation_tokens` from CC's cost telemetry, compute rolling median, and emit a loud warning at session start if the last session's token burn is >2σ above the rolling median for the same scenario class. Prove-It: seed with a synthetic spike, confirm warning fires. Value: catches silent CC-side regressions (caching bugs, prompt-inflation defaults) that we otherwise only notice from the invoice |
+| 221 | Post-mortem 2026-04-23 wizard integration | Three lessons from the 2026-04-23 post-mortem to fold into wizard docs. (a) **Explicit effort** — the incident had CC flipping reasoning_effort defaults (high → medium → xhigh/high). Our `model-effort-check.sh` already nudges to `xhigh`/`max`, but the wizard should cite this post-mortem as independent third-party evidence in `CLAUDE_CODE_SDLC_WIZARD.md` → Recommended Effort section: "never rely on the CC default, set it yourself." (b) **Extended-thinking + caching + idle sessions** — document the failure mode (context pruning during idle + cached prompts → dropped thinking blocks); add to SDLC.md troubleshooting or a new "Known CC gotchas" section. (c) **Prompt brevity constraints can compound** — the post-mortem describes a length-limit prompt change (including a line like `"keep text between tool calls to ≤25 words"` among other brevity edits) where ablation later measured a ~3% drop on one evaluation; the post-mortem attributes the drop to the broader length-limit prompt change, not to that single sentence alone. Audit our skills/hooks for verbosity caps: grep all SKILL.md and hook stdout for `≤\d+ words`, `<\d+ words`, `keep.*brief`, `be concise`, etc., decide case-by-case. Zero-API docs + grep audit, ~1 hr |
+| 222 | Prompt-compounding audit harness | Generalizes (c) from #221. Post-mortem 2026-04-23 showed isolated prompt changes can compound non-linearly across reasoning chains. Our wizard has ~40 prompt-injection sites (5 hooks × N stdout lines, 4 skills × instructions, CI PR review prompt, eval-criteria prompt). A small benign-looking constraint in any one could be silently capping quality. Proposal: periodic A/B benchmark comparing full-wizard harness vs. wizard-with-each-prompt-blanked, using existing E2E scoreboard. Flag any prompt whose removal *improves* score — that's a compounding constraint. Not urgent (needs API budget), stacks with #214 adaptive-thinking work once that's settled |
 | 218 | Evaluate MCP-tool hooks (CC 2.1.118) | CC 2.1.118 introduced `type: "mcp_tool"` — hooks can now directly invoke MCP tools instead of bash scripts. Our 5 current hooks are all bash (`_find-sdlc-root.sh`, `sdlc-prompt-check.sh`, `tdd-pretool-check.sh`, `instructions-loaded-check.sh`, `model-effort-check.sh`, `precompact-seam-check.sh`). Audit whether any would benefit: candidate is a hook that wants to read/write structured state (e.g. a "score-history reader" hook that queries persisted JSON via an MCP tool instead of cat+jq). Prove-It Gate applies: don't rewrite a working bash hook as an MCP call unless quality improves. Default answer is leave them — bash hooks are portable to Codex/OpenCode; MCP hooks may not be |
 | 219 | Re-verify #198 model-pin guidance against CC 2.1.117 persistence change | CC 2.1.117 changelog: "Model selections now persist across restarts despite project-level pins." Our #198 fix removed the top-level `"model": "opus[1m]"` from `.claude/settings.json` because it disables CC auto-mode. Verify the new persistence behavior doesn't break or change that recommendation: does "persist across restarts" mean session-picked model is now remembered (good, orthogonal), or does it mean project pins are re-asserted across restarts (could conflict with our migration path)? Test by removing the pin, picking Opus 4.7 mid-session, restarting CC, checking which model is active. Zero-API, ~10 min |
 | 223 | Adopt GPT-5.5 in review-tier guidance | **Status 2026-04-23:** GPT-5.5 confirmed released by OpenAI ([announcement](https://openai.com/index/introducing-gpt-5-5/)). **Two variants:** standard GPT-5.5 ($5/$30 per MTok input/output, rolling out to Plus/Pro/Business/Enterprise in ChatGPT **and Codex**) and GPT-5.5 Pro ($30/$180 per MTok, **ChatGPT-only** per rollout notes — API availability unclear on announcement day). For our Codex-CLI-backed cross-model review, **standard 5.5 is the usable ceiling** — Pro isn't in Codex. Pricing vs Opus 4.7 ($5/$25): GPT-5.5 is a 20% output premium; GPT-5.5 Pro is 6× standard — reserve for release-blocker / critical-infra one-offs. Supports xhigh reasoning effort, matches GPT-5.4 per-token latency, "especially strong in agentic coding, computer use... holding context across large systems, reasoning through ambiguous failures." Senior-engineer feedback quoted in the announcement says GPT-5.5 is "noticeably stronger than GPT-5.4 **and Claude Opus 4.7** at reasoning and autonomy" — if that holds in our harness, it changes our tiered-model strategy meaningfully. Codex CLI v0.124.0 (shipped the same day) does **not** mention 5.5 in release notes yet — needs either a CLI bump or `-c model="gpt-5.5"` override to verify availability. **Actions:** (a) confirm 5.5 is actually selectable from our current Codex CLI (try `codex exec -c model="gpt-5.5" ...` on a small prompt); (b) if yes, run the calibration: replay 3-5 historical `.reviews/*` handoffs with 5.5 xhigh and compare P1 counts vs the 5.4-xhigh reviews already on file — prove-it before we recommend switching; (c) if 5.5 is significantly stronger than Opus 4.7 on review work, revisit tiered-model roles: maybe Opus 4.7 codes + **GPT-5.5 reviews** for Complex (current plan), but also consider GPT-5.5 as executor for Light paths (replacing 5.4-mini) if latency/cost match; (d) update `CLAUDE_CODE_SDLC_WIZARD.md` cross-model-review section and SDLC skill's default codex-exec command once calibration confirms. Calibration replay cost ≈ $3-5 (a few handoffs × xhigh review). Blocker: CLI-side 5.5 availability |