Skip to content

fix(ci): #215 Tier 2 persist-scores gate uses real step output - #214

Merged
BaseInfinity merged 4 commits into
mainfrom
fix-215-tier2-dead-gate
Apr 23, 2026
Merged

fix(ci): #215 Tier 2 persist-scores gate uses real step output#214
BaseInfinity merged 4 commits into
mainfrom
fix-215-tier2-dead-gate

Conversation

@BaseInfinity

Copy link
Copy Markdown
Owner

Summary

Closes ROADMAP #215. Regression from #193.

The Tier 2 "Persist scores to PR branch" step at ci.yml:1436 was gated on steps.check-baseline.outputs.should_simulate, but the Tier 2 check-baseline step (line 1019) only emits has_baseline. The step had been silently dead — score-history.jsonl never got appended from Tier 2.

Changes

  • .github/workflows/ci.yml:1438 — change if-gate from should_simulatehas_baseline (1 word)
  • tests/test-workflow-triggers.sh — new regression test test_ci_gated_expressions_reference_real_outputs. Parses ci.yml with PyYAML, walks every step, builds a per-job map of step_id → emitted output names (matches both NAME=value and heredoc NAME<<EOF), then scans every steps.X.outputs.Y reference in if:, with:, env:, run:, outputs: and flags any that don't resolve.

Why the test is worth it

A single-line if-gate bug silently disabled a whole CI feature for weeks. The test is cheap (one Python script via python3+yaml, a standard test dep) and covers the entire file — not just #215. Regressions in any step's output wiring will trip it.

Test plan

  • Before fix: test reports exactly one broken ref (e2e-full-evaluation:step[17](Persist scores to PR branch).if -> should_simulate)
  • After fix: bash tests/test-workflow-triggers.sh → 164/0 pass
  • CI green

Follow-up captured separately

  • [ROADMAP] Anthropic 2026-04-23 post-mortem audit — user flagged for review, tracked as new roadmap item(s)

@github-actions

Copy link
Copy Markdown
Contributor

E2E Quick Check (Tier 1) ✅

Fast quality gate - single comparison per commit.

Scenario: hard-refactor

Layer Metric Value
L1: Model External Benchmark 58 (-20.0% vs baseline)
L2: SDLC Baseline (main) 10 / 10
Candidate (PR) 10 / 10
SDP (adjusted) 12.0 / 10
Delta +0.0
Combined Robustness 1.0
Status UNCHANGED

Result: No change in SDLC compliance (stable)

Interpretation: MODEL_DEGRADED

Criteria Breakdown
Criterion Score Evidence
🟢 plan_mode_outline 1/1 Agent explicitly outlined a plan before coding: 'Confidence: MEDIUM...' followed by a numbered Plan: with 5 steps (1. Update tests, 2. Run tests expect failures, 3. Implement Calculator class, 4. Run tests expect pass, 5. Self-review), and then created a TodoWrite list mirroring these steps.
🟢 plan_mode_tool 1/1 Agent called TodoWrite with 5 tasks (Update tests, Run tests RED, Implement Calculator, Run tests GREEN, Self-review) before making code edits, and updated the todo list throughout execution.
🟢 tdd_green_ran 1/1 Jest test output shown: 'PASS tests/app.test.js', 'PASS tests/utils.test.js', 'Test Suites: 2 passed, 2 total', 'Tests: 24 passed, 24 total'
🟢 tdd_green_pass 1/1 Final test run: 'Test Suites: 2 passed, 2 total / Tests: 24 passed, 24 total'
🟢 self_review 1/1 Agent used Read tool on both modified files (src/app.js and tests/app.test.js) after implementation as an explicit self-review step, then summarized the verified contents: 'Both files look correct. Self-review confirms...'
🟢 clean_code 1/1 Agent followed a single TDD approach: updated tests → verified RED (3 failures) → implemented Calculator class → verified GREEN (24 passing) → self-reviewed. No abandoned code, no commented-out dead code, no approach switches. Final files are clean with Calculator class and updated exports.
🟢 task_tracking 1/1 Found TodoWrite usage
🟢 confidence 1/1 Stated MEDIUM confidence
🟢 tdd_red 2/2 Test file created/edited before implementation file (or test-only task)
Historical Context

This scenario avg: 10.0 (1 runs)
Weakest criterion: tdd_red (77%)

Add merge-ready label for full 5x evaluation before merge.


Tier 1: 1x run each. SDP adjusts for external model conditions.

@github-actions

github-actions Bot commented Apr 23, 2026

Copy link
Copy Markdown
Contributor

Claude finished @BaseInfinity's task in 2m 31s —— View job


PR Code Review

Summary

One-word fix (should_simulatehas_baseline) that restores the Tier 2 "Persist scores to PR branch" step, which had been silently dead since #193 because Tier 2's check-baseline only emits has_baseline. Ships with a systemic regression test that parses ci.yml and flags any steps.<id>.outputs.<name> reference that doesn't resolve to an emitted output.

Recommendation: APPROVE

Verification

  • .github/workflows/ci.yml:1019-1027 — Tier 2 check-baseline emits only has_baseline (two branches, both cases).
  • .github/workflows/ci.yml:267-284 — Tier 1 check-baseline emits both has_baseline and should_simulate; all other should_simulate refs live in Tier 1's job scope (lines 287-985), so Tier 1 is unaffected.
  • After fix: every should_simulate reference now lives in a job whose check-baseline actually emits it. Every Tier 2 reference (1041-1504) uses has_baseline. Clean.
  • Gating on has_baseline is semantically correct for this step: all upstream simulation + score-recording steps in Tier 2 are already gated on has_baseline == 'true', so in the bootstrap case there's nothing new to persist anyway. Fork guard (head.repo.full_name == github.repository) is preserved. No security regression.

Findings

P0 — Critical (must fix before merge)

None.

P1 — Must Fix (should fix before merge)

None.

P2 — Suggestions (nice to have)

  • tests/test-workflow-triggers.sh:3289OUT_RE = r'echo\s+"?([A-Za-z_][A-Za-z0-9_]*)(=|<<)' won't match output names containing hyphens (e.g., echo "scenario-name=..."). Current repo convention is snake_case so this is fine today, but if someone ever emits a hyphenated key the test will flag it as a false positive. Trivial to future-proof by widening to [A-Za-z_][A-Za-z0-9_-]* — the REF_RE on the next line already accepts hyphens, so they're asymmetric. No action required for this PR.
  • tests/test-workflow-triggers.sh:3285 — The test only validates step ids whose run block writes to $GITHUB_OUTPUT. Outputs emitted by uses: composite/third-party actions are intentionally skipped, which is the right call (no metadata to introspect), but it means a dead gate on, say, steps.my-action.outputs.foo where the action doesn't emit foo would slip through. Acceptable trade-off and already implied by the design — just worth a one-line comment in the test so a future reader doesn't expand scope by accident.
  • tests/test-workflow-triggers.sh:3283 (doc) — Consider adding a one-line comment noting the scope ("validates shell-emitted outputs only; action outputs are out of scope") near the top of the test function so the limitation is visible without reading the regex.

Strengths

E2E Coverage Awareness

Change is in .github/workflows/ (CI/auto-update behavior). The Tier 2 persistence path isn't covered by an E2E scenario in tests/e2e/scenarios/ — the new static test in tests/test-workflow-triggers.sh is the right instrument here (structural correctness vs. behavioral E2E), so no scenario gap to flag.

Files Reviewed

  • .github/workflows/ci.yml (1-word gate fix)
  • tests/test-workflow-triggers.sh (new test_ci_gated_expressions_reference_real_outputs)
  • tests/e2e/score-history.jsonl (auto-appended Tier 1 score — no review concern)

  • Read PR diff
  • Read CLAUDE.md and CODE_REVIEW_EXCEPTIONS.md
  • Verify the fix against both check-baseline step definitions (lines 267, 1019)
  • Review regression test for correctness, scope, and security
  • Post final review
    Branch: fix-215-tier2-dead-gate

@github-actions

Copy link
Copy Markdown
Contributor

Review posted. Summary:

  • Recommendation: APPROVE — no P0 or P1 findings.
  • Verified the fix: Tier 2's check-baseline (line 1019) emits only has_baseline, Tier 1's (line 267) emits both, so gating Tier 2's "Persist scores" on has_baseline is both correct and the minimum fix.
  • The new test_ci_gated_expressions_reference_real_outputs test is well-scoped (per-job output map, handles both NAME= and heredoc NAME<< emission syntax) and catches this entire class of bug, not just roadmap(#220-222): Anthropic 2026-04-23 post-mortem learnings #215.
  • Three P2 suggestions around regex hyphen-handling and a scope-comment for the test — none blocking.

@BaseInfinity
BaseInfinity merged commit 2f88eb9 into main Apr 23, 2026
3 checks passed
@BaseInfinity
BaseInfinity deleted the fix-215-tier2-dead-gate branch April 23, 2026 19:32
@BaseInfinity BaseInfinity mentioned this pull request Apr 23, 2026
4 tasks
BaseInfinity added a commit that referenced this pull request Apr 23, 2026
…re for local-Max shepherd (#223)

* roadmap(#212): promote to TOP PRIORITY — tri-split billing architecture

Today's live-fire shepherding session hit the Anthropic API credit
cap mid-release (PR #222 e2e-quick-check failed 'Credit balance is
too low'). ~20+ dollars of API burn in one day across 12 PRs just
for E2E simulations via claude-code-action@v1.

Promoted to Next Up #0 (top of v1.37.0 queue).

Architectural refinement: tri-split billing.
- Simulation: claude --print on Max subscription ($0)
- Cross-model review of PR diffs: codex exec xhigh on OpenAI ($3-5)
- Orchestration: this Claude session on Max quota ($0)

The simulation MUST run on Claude because it's testing wizard
behavior on Claude. Running on GPT would test wizard portability,
not SDLC enforcement.

Prove-It Gate preserved: ≥3 PR score parity via overlapping
95% CI (not byte-equality) before trusting local signal.

Also flagged: #214 adaptive-thinking A/B should gate on #212 since
running it via paid API would burn another $12 per calibration.

* roadmap(#212): Codex-hardened after cross-model review (3/10 → addressed)

Codex xhigh review of the original #212 plan
(.reviews/latest-review-pr223.md) flagged 5 must-fix issues:

P0 — Trust boundary collapse: local shepherd exposes maintainer auth
     to fork PR code. Fix: restrict to trusted same-repo-author PRs
     only at first rollout. External contributors stay on CI-API.

P1 — 'Zero-API' is false: evaluate.sh calls Anthropic per-criterion
     during scoring. Moving only the simulation eliminates ~half the
     spend. Scoped honestly: 'partial-API' in title. Evaluator
     migration tracked as #228.

P1 — Prove-It gate too weak: '≥3 PRs overlapping 95% CI' can certify
     judge-noise parity while missing execution-path drift (Tier 2
     re-scores one transcript per #226). New gate: paired end-to-end
     on ≥5 scenarios × ≥3 runs each, then verify no statistically
     significant mean shift in ≥10 PRs post-migration.

P1 — Doesn't unblock merges: branch protection requires check-run,
     sticky PR comment ≠ check-run. Must POST a check via gh api.

P1 — Parity claim narrow: same model+prompt != same execution path.
     Added provenance fields to score-history rows (execution_path,
     host_os, cli_version, auth_mode).

P2 — Billing honesty: renamed from 'zero-API' to 'partial-API'.

Also added #228: evaluator API migration follow-up.
…aseline, not should_simulate)

Regression from #193. The Tier 2 'Persist scores to PR branch' step
was gated on steps.check-baseline.outputs.should_simulate, but the
Tier 2 check-baseline step only emits has_baseline. The step had been
silently dead — score-history.jsonl never got appended from Tier 2.

Fix:
- Change the if: gate to steps.check-baseline.outputs.has_baseline
- Add regression test that parses ci.yml and asserts every
  steps.<id>.outputs.<name> reference resolves to a real emitted
  output (same-job scope). Catches heredoc outputs (NAME<<EOF) as
  well as NAME=value. Regex tightened after initial false-positives.
…ntic-ai-sdlc-wizard into fix-215-tier2-dead-gate

# Conflicts:
#	tests/e2e/score-history.jsonl
BaseInfinity added a commit that referenced this pull request Apr 30, 2026
Implements EXECUTION PLAN from Codex strategic-priority review
(.reviews/grouping-review.md). Single zero-API hygiene release.

Fixed (#211 historical):
- Backfilled 5 corrupted rows in tests/e2e/score-history.jsonl
  (lines 22-25 + 30) from max_score:10 to max_score:11. UI scenarios
  with design_system criterion get an 11th point; the live writer
  was already correct (PR #214, v1.36.0) but the historical data
  remained corrupted. Codex-verified all 5 rows have
  criteria.design_system == true and remain valid JSON.

Closed paperwork-stale (already shipped, table rows just stale):
- #207 community scanner (shipped v1.39.0 + v1.56.0)
- #215 Tier 2 dead persist step (fixed v1.36.0; jobs later deleted)
- #217 model-effort-check loud warning (shipped 2026-04-24)
- #78 firmware E2E fixture
- #79 domain-adaptive testing diamond
- #80 SDLC effectiveness scoreboard

Verified (#219 doc-only):
- CC 2.1.118 local / 2.1.123 npm latest. Both settings.json files
  have no model key (jq verified). #198 recommendation unchanged.
- Optional manual UX check noted in roadmap row.

Codex round 1 CERTIFIED 9/10. Non-blocking P2 (stale response.json
from prior review) cleaned up.

No code changes outside the score-history backfill — pure roadmap
hygiene. Reduces backlog noise so future "what's next" reads honestly.
BaseInfinity added a commit that referenced this pull request May 5, 2026
User audit: "with all the opus fixes i dont think we need to do
adaptive thinking test anymore... im confused why does that [5.5
calibration] need API and replay harness huh this... we should audit
[weekly/monthly] when replacing them if we even need it."

Closed:
- #214 adaptive thinking A/B: moot given xhigh/max floor mandate.
  Saves $12 + spares API burn.
- #213 ship DISABLE_ADAPTIVE_THINKING default: same logic. xhigh
  floor > default band-aid env var. Keeps as opt-in for power users.
- #223 GPT-5.5 in review tier: already shipped via Codex config
  (model = "gpt-5.5" default). Formal calibration was academic.

Updated:
- #230 shepherd baseline/candidate: marked P3 nice-to-have with user
  quote "just get feature parity once." Not urgent.
- #231 weekly/monthly cleanup: audit outcome embedded. weekly-update
  actively useful (migrate its 9 API blocks). monthly-research mostly
  fails on cron + rarely run manually (last success 2026-03-27) +
  519 lines of low-value research-issue creation → proposed deletion
  instead of migration.

No code changes. doc-consistency green.
BaseInfinity added a commit that referenced this pull request May 5, 2026
Implements EXECUTION PLAN from Codex strategic-priority review
(.reviews/grouping-review.md). Single zero-API hygiene release.

Fixed (#211 historical):
- Backfilled 5 corrupted rows in tests/e2e/score-history.jsonl
  (lines 22-25 + 30) from max_score:10 to max_score:11. UI scenarios
  with design_system criterion get an 11th point; the live writer
  was already correct (PR #214, v1.36.0) but the historical data
  remained corrupted. Codex-verified all 5 rows have
  criteria.design_system == true and remain valid JSON.

Closed paperwork-stale (already shipped, table rows just stale):
- #207 community scanner (shipped v1.39.0 + v1.56.0)
- #215 Tier 2 dead persist step (fixed v1.36.0; jobs later deleted)
- #217 model-effort-check loud warning (shipped 2026-04-24)
- #78 firmware E2E fixture
- #79 domain-adaptive testing diamond
- #80 SDLC effectiveness scoreboard

Verified (#219 doc-only):
- CC 2.1.118 local / 2.1.123 npm latest. Both settings.json files
  have no model key (jq verified). #198 recommendation unchanged.
- Optional manual UX check noted in roadmap row.

Codex round 1 CERTIFIED 9/10. Non-blocking P2 (stale response.json
from prior review) cleaned up.

No code changes outside the score-history backfill — pure roadmap
hygiene. Reduces backlog noise so future "what's next" reads honestly.
BaseInfinity added a commit that referenced this pull request May 5, 2026
#206)

* feat(hooks): self-healing PreCompact on merged-PR stale handoff (#209)

Bug hit live 2026-04-19 after PR #205 merged — `.reviews/handoff.json`
stayed at PENDING_RECHECK, every subsequent /compact got blocked by the
user's own stale review artifact. Ships to consumers via CLI + plugin,
so every adopter of the handoff protocol who forgets to flip status
after merge will hit the same wall.

Fix: when status is PENDING_REVIEW/PENDING_RECHECK, parse optional
pr_number from handoff. If present AND gh is available, query
`gh pr view <pr_number> --json state` — MERGED unblocks (implicit
CERTIFIED). Missing pr_number, missing gh, offline, or any error
falls through to existing block (safe default).

4 new tests with mocked gh binary: merged unblocks, open blocks,
no pr_number blocks, gh-errors blocks. Hook suite 106 → 110.

Codex xhigh design review ran before implementation (verdict:
RECOMMENDED_ALTERNATIVE: 2, priority 9/10). Branch-awareness
alternative rejected — false-unblocks trunk-based workflows.

* chore: record E2E score [skip ci]

* test(hooks): add zero-stderr + gh-missing assertions (PR #206 Codex R1)

* chore: record E2E score [skip ci]

* docs(sdlc,ci): require Codex xhigh audit on CI logs in shepherd loop

* chore: record E2E score [skip ci]

* docs(roadmap): file #210 Node24 false-green + #211 tier1 11/10 (Codex CI-log audit on #206)

* chore: record E2E score [skip ci]

* docs(roadmap): add #212 local-Max E2E shepherd (zero-API alt)

* docs(sdlc): run Codex audit on Tier 1 AND Tier 2 CI logs separately

* docs(roadmap): #213 CLI template env-block gap — adaptive thinking + autocompact vars documented but not shipped

* docs(roadmap): #214 Prove-It A/B for adaptive thinking; gate #213 on result

* docs(roadmap): #215 Tier 2 persist step is dead code (Codex Tier 2 audit on #206)

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
BaseInfinity added a commit that referenced this pull request May 5, 2026
Token bloat audit (zero-API): inventory every file loaded on session
start, rank by token count, flag >5k trim candidates. Verify SDLC
BASELINE block printing 2x per UserPromptSubmit (likely duplicate
hook registration). Isolates repo-side bloat from Opus 4.7 model-side
burn (covered by #214).

OpenCode + local-LLM portability (strategic): reduce Anthropic
single-vendor dependency. Phase A: OpenCode running locally. Phase B:
local-LLM hardware scout (gaming laptop first, then $200-400 rig or
cloud-GPU). Phase C: opencode-sdlc-wizard sibling following the
existing pattern (agentic writes .claude/, codex writes .codex/,
new sibling writes .opencode/).
BaseInfinity added a commit that referenced this pull request May 5, 2026
* chore: record E2E score [skip ci]

* fix(ci): #215 Tier 2 persist-scores gate uses real step output (has_baseline, not should_simulate)

Regression from #193. The Tier 2 'Persist scores to PR branch' step
was gated on steps.check-baseline.outputs.should_simulate, but the
Tier 2 check-baseline step only emits has_baseline. The step had been
silently dead — score-history.jsonl never got appended from Tier 2.

Fix:
- Change the if: gate to steps.check-baseline.outputs.has_baseline
- Add regression test that parses ci.yml and asserts every
  steps.<id>.outputs.<name> reference resolves to a real emitted
  output (same-job scope). Catches heredoc outputs (NAME<<EOF) as
  well as NAME=value. Regex tightened after initial false-positives.

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
BaseInfinity added a commit that referenced this pull request May 5, 2026
BaseInfinity added a commit that referenced this pull request May 5, 2026
…re for local-Max shepherd (#223)

* roadmap(#212): promote to TOP PRIORITY — tri-split billing architecture

Today's live-fire shepherding session hit the Anthropic API credit
cap mid-release (PR #222 e2e-quick-check failed 'Credit balance is
too low'). ~20+ dollars of API burn in one day across 12 PRs just
for E2E simulations via claude-code-action@v1.

Promoted to Next Up #0 (top of v1.37.0 queue).

Architectural refinement: tri-split billing.
- Simulation: claude --print on Max subscription ($0)
- Cross-model review of PR diffs: codex exec xhigh on OpenAI ($3-5)
- Orchestration: this Claude session on Max quota ($0)

The simulation MUST run on Claude because it's testing wizard
behavior on Claude. Running on GPT would test wizard portability,
not SDLC enforcement.

Prove-It Gate preserved: ≥3 PR score parity via overlapping
95% CI (not byte-equality) before trusting local signal.

Also flagged: #214 adaptive-thinking A/B should gate on #212 since
running it via paid API would burn another $12 per calibration.

* roadmap(#212): Codex-hardened after cross-model review (3/10 → addressed)

Codex xhigh review of the original #212 plan
(.reviews/latest-review-pr223.md) flagged 5 must-fix issues:

P0 — Trust boundary collapse: local shepherd exposes maintainer auth
     to fork PR code. Fix: restrict to trusted same-repo-author PRs
     only at first rollout. External contributors stay on CI-API.

P1 — 'Zero-API' is false: evaluate.sh calls Anthropic per-criterion
     during scoring. Moving only the simulation eliminates ~half the
     spend. Scoped honestly: 'partial-API' in title. Evaluator
     migration tracked as #228.

P1 — Prove-It gate too weak: '≥3 PRs overlapping 95% CI' can certify
     judge-noise parity while missing execution-path drift (Tier 2
     re-scores one transcript per #226). New gate: paired end-to-end
     on ≥5 scenarios × ≥3 runs each, then verify no statistically
     significant mean shift in ≥10 PRs post-migration.

P1 — Doesn't unblock merges: branch protection requires check-run,
     sticky PR comment ≠ check-run. Must POST a check via gh api.

P1 — Parity claim narrow: same model+prompt != same execution path.
     Added provenance fields to score-history rows (execution_path,
     host_os, cli_version, auth_mode).

P2 — Billing honesty: renamed from 'zero-API' to 'partial-API'.

Also added #228: evaluator API migration follow-up.
BaseInfinity added a commit that referenced this pull request May 5, 2026
Implements EXECUTION PLAN from Codex strategic-priority review
(.reviews/grouping-review.md). Single zero-API hygiene release.

Fixed (#211 historical):
- Backfilled 5 corrupted rows in tests/e2e/score-history.jsonl
  (lines 22-25 + 30) from max_score:10 to max_score:11. UI scenarios
  with design_system criterion get an 11th point; the live writer
  was already correct (PR #214, v1.36.0) but the historical data
  remained corrupted. Codex-verified all 5 rows have
  criteria.design_system == true and remain valid JSON.

Closed paperwork-stale (already shipped, table rows just stale):
- #207 community scanner (shipped v1.39.0 + v1.56.0)
- #215 Tier 2 dead persist step (fixed v1.36.0; jobs later deleted)
- #217 model-effort-check loud warning (shipped 2026-04-24)
- #78 firmware E2E fixture
- #79 domain-adaptive testing diamond
- #80 SDLC effectiveness scoreboard

Verified (#219 doc-only):
- CC 2.1.118 local / 2.1.123 npm latest. Both settings.json files
  have no model key (jq verified). #198 recommendation unchanged.
- Optional manual UX check noted in roadmap row.

Codex round 1 CERTIFIED 9/10. Non-blocking P2 (stale response.json
from prior review) cleaned up.

No code changes outside the score-history backfill — pure roadmap
hygiene. Reduces backlog noise so future "what's next" reads honestly.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant