roadmap(#220-222): Anthropic 2026-04-23 post-mortem learnings by BaseInfinity · Pull Request #215 · BaseInfinity/claude-sdlc-harness

BaseInfinity · 2026-04-23T18:30:03Z

Summary

Three backlog items from the Anthropic 2026-04-23 post-mortem (anthropic.com/engineering/april-23-postmortem).

roadmap(#224-227): follow-ups from Codex review of auto-PRs #220 — Token-spike anomaly detection. The post-mortem's caching bug ("continuously dropped thinking blocks from subsequent requests") was only visible as usage drain. Our effectiveness scoreboard (fix(ci): ensure needs-regression-test label exists before adding #80) doesn't track token burn. Add a per-session token-delta signal with >2σ rolling-median anomaly detection.
fix(ci): add workflow_dispatch fallback to release.yml #221 — Fold 3 concrete lessons into wizard docs: (a) explicit effort (post-mortem independently validates our model-effort-check.sh stance), (b) extended-thinking + caching + idle-session failure mode (document as gotcha), (c) verbosity caps can cost intelligence (a single 25-word constraint caused a 3% drop in their incident). Includes a grep audit of our SKILL.md and hook stdout for verbosity constraints.
release: v1.36.1 — slug migration + npm pkg fix + gitignore cruft #222 — Prompt-compounding audit harness. Generalizes (c): A/B blank each of our ~40 prompt-injection sites and measure impact. Needs API budget; stacks with fix(ci): #215 Tier 2 persist-scores gate uses real step output #214.

Why PR

Branch protection blocks direct push to main.

Test plan

CI green (doc-only roadmap addition)

github-actions · 2026-04-23T18:30:18Z

Claude Code Review Summary

Status: Skipped (trivial changes)

This PR only contains documentation or configuration changes. Code review skipped to save API costs.

To force a review, add the needs-review label.

github-actions · 2026-04-23T18:34:29Z

E2E Quick Check (Tier 1) ✅

Fast quality gate - single comparison per commit.

Scenario: medium-add-feature

Layer	Metric	Value
L1: Model	External Benchmark	58 (-20.0% vs baseline)
L2: SDLC	Baseline (main)	10 / 10
	Candidate (PR)	10 / 10
	SDP (adjusted)	12.0 / 10
	Delta	+0.0
Combined	Robustness	1.0
	Status	UNCHANGED

Result: No change in SDLC compliance (stable)

Interpretation: MODEL_DEGRADED

Criteria Breakdown

	Criterion	Score	Evidence
🟢	plan_mode_outline	1/1	Agent created a TodoWrite plan with 4 numbered steps before implementing: 1) Write failing test for validateEmail (TDD red phase), 2) Implement validateEmail in src/app.js, 3) Run npm test to verify all tests pass, 4) Self-review modified files. Also stated 'I'll start by reading the scenario file and setting up task tracking.'
🟢	plan_mode_tool	1/1	Agent called TodoWrite with 4 tasks (write failing test, implement, run tests, self-review) before any Edit tool calls
🟢	tdd_green_ran	1/1	Test runner output shows 'PASS tests/app.test.js', 'PASS tests/utils.test.js', 'Test Suites: 2 passed, 2 total', 'Tests: 32 passed, 32 total' after implementation (green phase).
🟢	tdd_green_pass	1/1	Final test run shows 'Test Suites: 2 passed, 2 total' and 'Tests: 32 passed, 32 total'
🟢	self_review	1/1	Agent used Read tool on both modified files after implementation: Read src/app.js at offset 114 (showing the validateEmail function) and Read tests/app.test.js at offset 113 (showing the new test block), then stated 'Self-review looks good — implementation is clean and minimal'
🟢	clean_code	1/1	Single coherent TDD flow: planned with TodoWrite, wrote failing tests, confirmed red phase (8 failures), implemented validateEmail, confirmed green phase (32 passing), self-reviewed. No abandoned code, no contradictory changes, no dead code.
🟢	task_tracking	1/1	Found TodoWrite usage
🟢	confidence	1/1	Stated HIGH confidence
🟢	tdd_red	2/2	Test file created/edited before implementation file (or test-only task)

Historical Context

This scenario avg: 10.0 (4 runs)
Weakest criterion: tdd_red (78%)

Add merge-ready label for full 5x evaluation before merge.

Tier 1: 1x run each. SDP adjusts for external model conditions.

Codex batch doc-review (batch-doc-prs-209-211-215-218, 2026-04-23) raised P1 on PR #215's #221(c) wording. Original claim overattributed the 3% drop to a single sentence. Post-mortem actually describes a broader length-limit prompt change with that sentence among other brevity edits; ablation measured ~3% drop on one evaluation attributed to the broader change. Fix: reword to attribute to the broader length-limit prompt change (not single sentence). Keeps the sentence as example of the class of change without overclaiming causation.

* chore: record E2E score [skip ci] * fix(ci): #215 Tier 2 persist-scores gate uses real step output (has_baseline, not should_simulate) Regression from #193. The Tier 2 'Persist scores to PR branch' step was gated on steps.check-baseline.outputs.should_simulate, but the Tier 2 check-baseline step only emits has_baseline. The step had been silently dead — score-history.jsonl never got appended from Tier 2. Fix: - Change the if: gate to steps.check-baseline.outputs.has_baseline - Add regression test that parses ci.yml and asserts every steps.<id>.outputs.<name> reference resolves to a real emitted output (same-job scope). Catches heredoc outputs (NAME<<EOF) as well as NAME=value. Regex tightened after initial false-positives. --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

# Conflicts: # ROADMAP.md # tests/e2e/score-history.jsonl

github-actions · 2026-04-23T19:33:15Z

PR Review in Progress

Read changed files
Check CLAUDE.md and CODE_REVIEW_EXCEPTIONS.md
Analyze changes vs. referenced issues roadmap(#224-227): follow-ups from Codex review of auto-PRs #220, fix(ci): add workflow_dispatch fallback to release.yml #221, release: v1.36.1 — slug migration + npm pkg fix + gitignore cruft #222
Post review

View job run

Three backlog items drawn from the post-mortem (anthropic.com/engineering/april-23-postmortem): #220: token-spike anomaly detection — our effectiveness scoreboard doesn't track token burn; post-mortem's caching bug was only visible as usage drain. #221: fold the 3 concrete lessons into wizard docs — explicit effort (validates our stance), extended-thinking + caching + idle sessions gotcha, and the 25-word-constraint intelligence drop. Includes a grep audit of our skills/hooks for verbosity caps. #222: prompt-compounding audit harness — generalize the verbosity-cap lesson to an A/B that blanks each prompt and measures impact.

Implements EXECUTION PLAN from Codex strategic-priority review (.reviews/grouping-review.md). Single zero-API hygiene release. Fixed (#211 historical): - Backfilled 5 corrupted rows in tests/e2e/score-history.jsonl (lines 22-25 + 30) from max_score:10 to max_score:11. UI scenarios with design_system criterion get an 11th point; the live writer was already correct (PR #214, v1.36.0) but the historical data remained corrupted. Codex-verified all 5 rows have criteria.design_system == true and remain valid JSON. Closed paperwork-stale (already shipped, table rows just stale): - #207 community scanner (shipped v1.39.0 + v1.56.0) - #215 Tier 2 dead persist step (fixed v1.36.0; jobs later deleted) - #217 model-effort-check loud warning (shipped 2026-04-24) - #78 firmware E2E fixture - #79 domain-adaptive testing diamond - #80 SDLC effectiveness scoreboard Verified (#219 doc-only): - CC 2.1.118 local / 2.1.123 npm latest. Both settings.json files have no model key (jq verified). #198 recommendation unchanged. - Optional manual UX check noted in roadmap row. Codex round 1 CERTIFIED 9/10. Non-blocking P2 (stale response.json from prior review) cleaned up. No code changes outside the score-history backfill — pure roadmap hygiene. Reduces backlog noise so future "what's next" reads honestly.

#206) * feat(hooks): self-healing PreCompact on merged-PR stale handoff (#209) Bug hit live 2026-04-19 after PR #205 merged — `.reviews/handoff.json` stayed at PENDING_RECHECK, every subsequent /compact got blocked by the user's own stale review artifact. Ships to consumers via CLI + plugin, so every adopter of the handoff protocol who forgets to flip status after merge will hit the same wall. Fix: when status is PENDING_REVIEW/PENDING_RECHECK, parse optional pr_number from handoff. If present AND gh is available, query `gh pr view <pr_number> --json state` — MERGED unblocks (implicit CERTIFIED). Missing pr_number, missing gh, offline, or any error falls through to existing block (safe default). 4 new tests with mocked gh binary: merged unblocks, open blocks, no pr_number blocks, gh-errors blocks. Hook suite 106 → 110. Codex xhigh design review ran before implementation (verdict: RECOMMENDED_ALTERNATIVE: 2, priority 9/10). Branch-awareness alternative rejected — false-unblocks trunk-based workflows. * chore: record E2E score [skip ci] * test(hooks): add zero-stderr + gh-missing assertions (PR #206 Codex R1) * chore: record E2E score [skip ci] * docs(sdlc,ci): require Codex xhigh audit on CI logs in shepherd loop * chore: record E2E score [skip ci] * docs(roadmap): file #210 Node24 false-green + #211 tier1 11/10 (Codex CI-log audit on #206) * chore: record E2E score [skip ci] * docs(roadmap): add #212 local-Max E2E shepherd (zero-API alt) * docs(sdlc): run Codex audit on Tier 1 AND Tier 2 CI logs separately * docs(roadmap): #213 CLI template env-block gap — adaptive thinking + autocompact vars documented but not shipped * docs(roadmap): #214 Prove-It A/B for adaptive thinking; gate #213 on result * docs(roadmap): #215 Tier 2 persist step is dead code (Codex Tier 2 audit on #206) --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* chore: record E2E score [skip ci] * fix(ci): #215 Tier 2 persist-scores gate uses real step output (has_baseline, not should_simulate) Regression from #193. The Tier 2 'Persist scores to PR branch' step was gated on steps.check-baseline.outputs.should_simulate, but the Tier 2 check-baseline step only emits has_baseline. The step had been silently dead — score-history.jsonl never got appended from Tier 2. Fix: - Change the if: gate to steps.check-baseline.outputs.has_baseline - Add regression test that parses ci.yml and asserts every steps.<id>.outputs.<name> reference resolves to a real emitted output (same-job scope). Catches heredoc outputs (NAME<<EOF) as well as NAME=value. Regex tightened after initial false-positives. --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* roadmap(#221): reword 3% intelligence drop attribution (Codex P1) Codex batch doc-review (batch-doc-prs-209-211-215-218, 2026-04-23) raised P1 on PR #215's #221(c) wording. Original claim overattributed the 3% drop to a single sentence. Post-mortem actually describes a broader length-limit prompt change with that sentence among other brevity edits; ablation measured ~3% drop on one evaluation attributed to the broader change. Fix: reword to attribute to the broader length-limit prompt change (not single sentence). Keeps the sentence as example of the class of change without overclaiming causation. * chore: record E2E score [skip ci] * roadmap(#220, #221, #222): Anthropic 2026-04-23 post-mortem learnings Three backlog items drawn from the post-mortem (anthropic.com/engineering/april-23-postmortem): #220: token-spike anomaly detection — our effectiveness scoreboard doesn't track token burn; post-mortem's caching bug was only visible as usage drain. #221: fold the 3 concrete lessons into wizard docs — explicit effort (validates our stance), extended-thinking + caching + idle sessions gotcha, and the 25-word-constraint intelligence drop. Includes a grep audit of our skills/hooks for verbosity caps. #222: prompt-compounding audit harness — generalize the verbosity-cap lesson to an A/B that blanks each prompt and measures impact. * chore: record E2E score [skip ci] --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…dit on #206)

Implements EXECUTION PLAN from Codex strategic-priority review (.reviews/grouping-review.md). Single zero-API hygiene release. Fixed (#211 historical): - Backfilled 5 corrupted rows in tests/e2e/score-history.jsonl (lines 22-25 + 30) from max_score:10 to max_score:11. UI scenarios with design_system criterion get an 11th point; the live writer was already correct (PR #214, v1.36.0) but the historical data remained corrupted. Codex-verified all 5 rows have criteria.design_system == true and remain valid JSON. Closed paperwork-stale (already shipped, table rows just stale): - #207 community scanner (shipped v1.39.0 + v1.56.0) - #215 Tier 2 dead persist step (fixed v1.36.0; jobs later deleted) - #217 model-effort-check loud warning (shipped 2026-04-24) - #78 firmware E2E fixture - #79 domain-adaptive testing diamond - #80 SDLC effectiveness scoreboard Verified (#219 doc-only): - CC 2.1.118 local / 2.1.123 npm latest. Both settings.json files have no model key (jq verified). #198 recommendation unchanged. - Optional manual UX check noted in roadmap row. Codex round 1 CERTIFIED 9/10. Non-blocking P2 (stale response.json from prior review) cleaned up. No code changes outside the score-history backfill — pure roadmap hygiene. Reduces backlog noise so future "what's next" reads honestly.

…335) Rows #78, #79, #80, #215, #217 each say in their own cell text: Stale duplicate of <canonical row>; <work> shipped. v1.62.0 paperwork close. The canonical entries live earlier in the file. Keeping the duplicates just inflates the unprioritized table without adding info — every field reduces to "see <other row>." Removing them shrinks the unprioritized table by 5 rows and removes the "wait, is this still open?" friction when scanning. doc-consistency tests 35/35 still green (no test asserts row count).

Rows #78, #79, #80, #215, #217 each say in their own cell text: Stale duplicate of <canonical row>; <work> shipped. v1.62.0 paperwork close. The canonical entries live earlier in the file. Keeping the duplicates just inflates the unprioritized table without adding info — every field reduces to "see <other row>." Removing them shrinks the unprioritized table by 5 rows and removes the "wait, is this still open?" friction when scanning. doc-consistency tests 35/35 still green (no test asserts row count).

BaseInfinity mentioned this pull request Apr 23, 2026

fix(ci): #215 Tier 2 persist-scores gate uses real step output #214

Merged

3 tasks

BaseInfinity mentioned this pull request Apr 23, 2026

fix(ci): #211 score-history records real max_score (not hardcoded 10) #216

Merged

5 tasks

github-actions Bot and others added 2 commits April 23, 2026 19:32

chore: record E2E score [skip ci]

5cd2d46

Merge branch 'main' into roadmap-postmortem-2026-04-23

7110108

# Conflicts: # ROADMAP.md # tests/e2e/score-history.jsonl

BaseInfinity merged commit 757aff4 into main Apr 23, 2026

BaseInfinity deleted the roadmap-postmortem-2026-04-23 branch April 23, 2026 19:33

BaseInfinity mentioned this pull request Apr 23, 2026

release: v1.36.0 #219

Merged

4 tasks

BaseInfinity and others added 2 commits April 23, 2026 17:39

chore: record E2E score [skip ci]

3c1b026

BaseInfinity mentioned this pull request Apr 30, 2026

chore: v1.62.0 — roadmap hygiene + #211 historical backfill #298

Merged

6 tasks

BaseInfinity added a commit that referenced this pull request May 5, 2026

docs(roadmap): #215 Tier 2 persist step is dead code (Codex Tier 2 au…

747b3e6

…dit on #206)

BaseInfinity mentioned this pull request May 7, 2026

gc(#236): drop 5 self-documented 'stale duplicate' rows from ROADMAP #335

Merged

Navigation Menu

Uh oh!

roadmap(#220-222): Anthropic 2026-04-23 post-mortem learnings - #215

roadmap(#220-222): Anthropic 2026-04-23 post-mortem learnings#215
BaseInfinity merged 5 commits into
mainfrom
roadmap-postmortem-2026-04-23

BaseInfinity commented Apr 23, 2026

Uh oh!

github-actions Bot commented Apr 23, 2026

Uh oh!

github-actions Bot commented Apr 23, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Apr 23, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

BaseInfinity commented Apr 23, 2026

Summary

Why PR

Test plan

Uh oh!

github-actions Bot commented Apr 23, 2026

Claude Code Review Summary

Uh oh!

github-actions Bot commented Apr 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

E2E Quick Check (Tier 1) ✅

Result: No change in SDLC compliance (stable)

Uh oh!

github-actions Bot commented Apr 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Review in Progress

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

github-actions Bot commented Apr 23, 2026 •

edited

Loading

github-actions Bot commented Apr 23, 2026 •

edited

Loading