fix(ci): #215 Tier 2 persist-scores gate uses real step output - #214
Conversation
E2E Quick Check (Tier 1) ✅Fast quality gate - single comparison per commit.
Result: No change in SDLC compliance (stable)
Criteria Breakdown
Historical ContextThis scenario avg: 10.0 (1 runs) Add Tier 1: 1x run each. SDP adjusts for external model conditions. |
|
Claude finished @BaseInfinity's task in 2m 31s —— View job PR Code ReviewSummaryOne-word fix ( Recommendation: APPROVEVerification
FindingsP0 — Critical (must fix before merge)None. P1 — Must Fix (should fix before merge)None. P2 — Suggestions (nice to have)
Strengths
E2E Coverage AwarenessChange is in Files Reviewed
|
|
Review posted. Summary:
|
…re for local-Max shepherd (#223) * roadmap(#212): promote to TOP PRIORITY — tri-split billing architecture Today's live-fire shepherding session hit the Anthropic API credit cap mid-release (PR #222 e2e-quick-check failed 'Credit balance is too low'). ~20+ dollars of API burn in one day across 12 PRs just for E2E simulations via claude-code-action@v1. Promoted to Next Up #0 (top of v1.37.0 queue). Architectural refinement: tri-split billing. - Simulation: claude --print on Max subscription ($0) - Cross-model review of PR diffs: codex exec xhigh on OpenAI ($3-5) - Orchestration: this Claude session on Max quota ($0) The simulation MUST run on Claude because it's testing wizard behavior on Claude. Running on GPT would test wizard portability, not SDLC enforcement. Prove-It Gate preserved: ≥3 PR score parity via overlapping 95% CI (not byte-equality) before trusting local signal. Also flagged: #214 adaptive-thinking A/B should gate on #212 since running it via paid API would burn another $12 per calibration. * roadmap(#212): Codex-hardened after cross-model review (3/10 → addressed) Codex xhigh review of the original #212 plan (.reviews/latest-review-pr223.md) flagged 5 must-fix issues: P0 — Trust boundary collapse: local shepherd exposes maintainer auth to fork PR code. Fix: restrict to trusted same-repo-author PRs only at first rollout. External contributors stay on CI-API. P1 — 'Zero-API' is false: evaluate.sh calls Anthropic per-criterion during scoring. Moving only the simulation eliminates ~half the spend. Scoped honestly: 'partial-API' in title. Evaluator migration tracked as #228. P1 — Prove-It gate too weak: '≥3 PRs overlapping 95% CI' can certify judge-noise parity while missing execution-path drift (Tier 2 re-scores one transcript per #226). New gate: paired end-to-end on ≥5 scenarios × ≥3 runs each, then verify no statistically significant mean shift in ≥10 PRs post-migration. P1 — Doesn't unblock merges: branch protection requires check-run, sticky PR comment ≠ check-run. Must POST a check via gh api. P1 — Parity claim narrow: same model+prompt != same execution path. Added provenance fields to score-history rows (execution_path, host_os, cli_version, auth_mode). P2 — Billing honesty: renamed from 'zero-API' to 'partial-API'. Also added #228: evaluator API migration follow-up.
…aseline, not should_simulate) Regression from #193. The Tier 2 'Persist scores to PR branch' step was gated on steps.check-baseline.outputs.should_simulate, but the Tier 2 check-baseline step only emits has_baseline. The step had been silently dead — score-history.jsonl never got appended from Tier 2. Fix: - Change the if: gate to steps.check-baseline.outputs.has_baseline - Add regression test that parses ci.yml and asserts every steps.<id>.outputs.<name> reference resolves to a real emitted output (same-job scope). Catches heredoc outputs (NAME<<EOF) as well as NAME=value. Regex tightened after initial false-positives.
…ntic-ai-sdlc-wizard into fix-215-tier2-dead-gate # Conflicts: # tests/e2e/score-history.jsonl
Implements EXECUTION PLAN from Codex strategic-priority review (.reviews/grouping-review.md). Single zero-API hygiene release. Fixed (#211 historical): - Backfilled 5 corrupted rows in tests/e2e/score-history.jsonl (lines 22-25 + 30) from max_score:10 to max_score:11. UI scenarios with design_system criterion get an 11th point; the live writer was already correct (PR #214, v1.36.0) but the historical data remained corrupted. Codex-verified all 5 rows have criteria.design_system == true and remain valid JSON. Closed paperwork-stale (already shipped, table rows just stale): - #207 community scanner (shipped v1.39.0 + v1.56.0) - #215 Tier 2 dead persist step (fixed v1.36.0; jobs later deleted) - #217 model-effort-check loud warning (shipped 2026-04-24) - #78 firmware E2E fixture - #79 domain-adaptive testing diamond - #80 SDLC effectiveness scoreboard Verified (#219 doc-only): - CC 2.1.118 local / 2.1.123 npm latest. Both settings.json files have no model key (jq verified). #198 recommendation unchanged. - Optional manual UX check noted in roadmap row. Codex round 1 CERTIFIED 9/10. Non-blocking P2 (stale response.json from prior review) cleaned up. No code changes outside the score-history backfill — pure roadmap hygiene. Reduces backlog noise so future "what's next" reads honestly.
User audit: "with all the opus fixes i dont think we need to do adaptive thinking test anymore... im confused why does that [5.5 calibration] need API and replay harness huh this... we should audit [weekly/monthly] when replacing them if we even need it." Closed: - #214 adaptive thinking A/B: moot given xhigh/max floor mandate. Saves $12 + spares API burn. - #213 ship DISABLE_ADAPTIVE_THINKING default: same logic. xhigh floor > default band-aid env var. Keeps as opt-in for power users. - #223 GPT-5.5 in review tier: already shipped via Codex config (model = "gpt-5.5" default). Formal calibration was academic. Updated: - #230 shepherd baseline/candidate: marked P3 nice-to-have with user quote "just get feature parity once." Not urgent. - #231 weekly/monthly cleanup: audit outcome embedded. weekly-update actively useful (migrate its 9 API blocks). monthly-research mostly fails on cron + rarely run manually (last success 2026-03-27) + 519 lines of low-value research-issue creation → proposed deletion instead of migration. No code changes. doc-consistency green.
Implements EXECUTION PLAN from Codex strategic-priority review (.reviews/grouping-review.md). Single zero-API hygiene release. Fixed (#211 historical): - Backfilled 5 corrupted rows in tests/e2e/score-history.jsonl (lines 22-25 + 30) from max_score:10 to max_score:11. UI scenarios with design_system criterion get an 11th point; the live writer was already correct (PR #214, v1.36.0) but the historical data remained corrupted. Codex-verified all 5 rows have criteria.design_system == true and remain valid JSON. Closed paperwork-stale (already shipped, table rows just stale): - #207 community scanner (shipped v1.39.0 + v1.56.0) - #215 Tier 2 dead persist step (fixed v1.36.0; jobs later deleted) - #217 model-effort-check loud warning (shipped 2026-04-24) - #78 firmware E2E fixture - #79 domain-adaptive testing diamond - #80 SDLC effectiveness scoreboard Verified (#219 doc-only): - CC 2.1.118 local / 2.1.123 npm latest. Both settings.json files have no model key (jq verified). #198 recommendation unchanged. - Optional manual UX check noted in roadmap row. Codex round 1 CERTIFIED 9/10. Non-blocking P2 (stale response.json from prior review) cleaned up. No code changes outside the score-history backfill — pure roadmap hygiene. Reduces backlog noise so future "what's next" reads honestly.
#206) * feat(hooks): self-healing PreCompact on merged-PR stale handoff (#209) Bug hit live 2026-04-19 after PR #205 merged — `.reviews/handoff.json` stayed at PENDING_RECHECK, every subsequent /compact got blocked by the user's own stale review artifact. Ships to consumers via CLI + plugin, so every adopter of the handoff protocol who forgets to flip status after merge will hit the same wall. Fix: when status is PENDING_REVIEW/PENDING_RECHECK, parse optional pr_number from handoff. If present AND gh is available, query `gh pr view <pr_number> --json state` — MERGED unblocks (implicit CERTIFIED). Missing pr_number, missing gh, offline, or any error falls through to existing block (safe default). 4 new tests with mocked gh binary: merged unblocks, open blocks, no pr_number blocks, gh-errors blocks. Hook suite 106 → 110. Codex xhigh design review ran before implementation (verdict: RECOMMENDED_ALTERNATIVE: 2, priority 9/10). Branch-awareness alternative rejected — false-unblocks trunk-based workflows. * chore: record E2E score [skip ci] * test(hooks): add zero-stderr + gh-missing assertions (PR #206 Codex R1) * chore: record E2E score [skip ci] * docs(sdlc,ci): require Codex xhigh audit on CI logs in shepherd loop * chore: record E2E score [skip ci] * docs(roadmap): file #210 Node24 false-green + #211 tier1 11/10 (Codex CI-log audit on #206) * chore: record E2E score [skip ci] * docs(roadmap): add #212 local-Max E2E shepherd (zero-API alt) * docs(sdlc): run Codex audit on Tier 1 AND Tier 2 CI logs separately * docs(roadmap): #213 CLI template env-block gap — adaptive thinking + autocompact vars documented but not shipped * docs(roadmap): #214 Prove-It A/B for adaptive thinking; gate #213 on result * docs(roadmap): #215 Tier 2 persist step is dead code (Codex Tier 2 audit on #206) --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Token bloat audit (zero-API): inventory every file loaded on session start, rank by token count, flag >5k trim candidates. Verify SDLC BASELINE block printing 2x per UserPromptSubmit (likely duplicate hook registration). Isolates repo-side bloat from Opus 4.7 model-side burn (covered by #214). OpenCode + local-LLM portability (strategic): reduce Anthropic single-vendor dependency. Phase A: OpenCode running locally. Phase B: local-LLM hardware scout (gaming laptop first, then $200-400 rig or cloud-GPU). Phase C: opencode-sdlc-wizard sibling following the existing pattern (agentic writes .claude/, codex writes .codex/, new sibling writes .opencode/).
* chore: record E2E score [skip ci] * fix(ci): #215 Tier 2 persist-scores gate uses real step output (has_baseline, not should_simulate) Regression from #193. The Tier 2 'Persist scores to PR branch' step was gated on steps.check-baseline.outputs.should_simulate, but the Tier 2 check-baseline step only emits has_baseline. The step had been silently dead — score-history.jsonl never got appended from Tier 2. Fix: - Change the if: gate to steps.check-baseline.outputs.has_baseline - Add regression test that parses ci.yml and asserts every steps.<id>.outputs.<name> reference resolves to a real emitted output (same-job scope). Catches heredoc outputs (NAME<<EOF) as well as NAME=value. Regex tightened after initial false-positives. --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
…re for local-Max shepherd (#223) * roadmap(#212): promote to TOP PRIORITY — tri-split billing architecture Today's live-fire shepherding session hit the Anthropic API credit cap mid-release (PR #222 e2e-quick-check failed 'Credit balance is too low'). ~20+ dollars of API burn in one day across 12 PRs just for E2E simulations via claude-code-action@v1. Promoted to Next Up #0 (top of v1.37.0 queue). Architectural refinement: tri-split billing. - Simulation: claude --print on Max subscription ($0) - Cross-model review of PR diffs: codex exec xhigh on OpenAI ($3-5) - Orchestration: this Claude session on Max quota ($0) The simulation MUST run on Claude because it's testing wizard behavior on Claude. Running on GPT would test wizard portability, not SDLC enforcement. Prove-It Gate preserved: ≥3 PR score parity via overlapping 95% CI (not byte-equality) before trusting local signal. Also flagged: #214 adaptive-thinking A/B should gate on #212 since running it via paid API would burn another $12 per calibration. * roadmap(#212): Codex-hardened after cross-model review (3/10 → addressed) Codex xhigh review of the original #212 plan (.reviews/latest-review-pr223.md) flagged 5 must-fix issues: P0 — Trust boundary collapse: local shepherd exposes maintainer auth to fork PR code. Fix: restrict to trusted same-repo-author PRs only at first rollout. External contributors stay on CI-API. P1 — 'Zero-API' is false: evaluate.sh calls Anthropic per-criterion during scoring. Moving only the simulation eliminates ~half the spend. Scoped honestly: 'partial-API' in title. Evaluator migration tracked as #228. P1 — Prove-It gate too weak: '≥3 PRs overlapping 95% CI' can certify judge-noise parity while missing execution-path drift (Tier 2 re-scores one transcript per #226). New gate: paired end-to-end on ≥5 scenarios × ≥3 runs each, then verify no statistically significant mean shift in ≥10 PRs post-migration. P1 — Doesn't unblock merges: branch protection requires check-run, sticky PR comment ≠ check-run. Must POST a check via gh api. P1 — Parity claim narrow: same model+prompt != same execution path. Added provenance fields to score-history rows (execution_path, host_os, cli_version, auth_mode). P2 — Billing honesty: renamed from 'zero-API' to 'partial-API'. Also added #228: evaluator API migration follow-up.
Implements EXECUTION PLAN from Codex strategic-priority review (.reviews/grouping-review.md). Single zero-API hygiene release. Fixed (#211 historical): - Backfilled 5 corrupted rows in tests/e2e/score-history.jsonl (lines 22-25 + 30) from max_score:10 to max_score:11. UI scenarios with design_system criterion get an 11th point; the live writer was already correct (PR #214, v1.36.0) but the historical data remained corrupted. Codex-verified all 5 rows have criteria.design_system == true and remain valid JSON. Closed paperwork-stale (already shipped, table rows just stale): - #207 community scanner (shipped v1.39.0 + v1.56.0) - #215 Tier 2 dead persist step (fixed v1.36.0; jobs later deleted) - #217 model-effort-check loud warning (shipped 2026-04-24) - #78 firmware E2E fixture - #79 domain-adaptive testing diamond - #80 SDLC effectiveness scoreboard Verified (#219 doc-only): - CC 2.1.118 local / 2.1.123 npm latest. Both settings.json files have no model key (jq verified). #198 recommendation unchanged. - Optional manual UX check noted in roadmap row. Codex round 1 CERTIFIED 9/10. Non-blocking P2 (stale response.json from prior review) cleaned up. No code changes outside the score-history backfill — pure roadmap hygiene. Reduces backlog noise so future "what's next" reads honestly.
Summary
Closes ROADMAP #215. Regression from #193.
The Tier 2 "Persist scores to PR branch" step at
ci.yml:1436was gated onsteps.check-baseline.outputs.should_simulate, but the Tier 2check-baselinestep (line 1019) only emitshas_baseline. The step had been silently dead —score-history.jsonlnever got appended from Tier 2.Changes
.github/workflows/ci.yml:1438— change if-gate fromshould_simulate→has_baseline(1 word)tests/test-workflow-triggers.sh— new regression testtest_ci_gated_expressions_reference_real_outputs. Parses ci.yml with PyYAML, walks every step, builds a per-job map of step_id → emitted output names (matches bothNAME=valueand heredocNAME<<EOF), then scans everysteps.X.outputs.Yreference inif:,with:,env:,run:,outputs:and flags any that don't resolve.Why the test is worth it
A single-line if-gate bug silently disabled a whole CI feature for weeks. The test is cheap (one Python script via python3+yaml, a standard test dep) and covers the entire file — not just #215. Regressions in any step's output wiring will trip it.
Test plan
e2e-full-evaluation:step[17](Persist scores to PR branch).if -> should_simulate)bash tests/test-workflow-triggers.sh→ 164/0 passFollow-up captured separately