Skip to content

feat: add optional cross-model review loop - #90

Merged
BaseInfinity merged 4 commits into
mainfrom
feat/cross-model-review
Mar 27, 2026
Merged

feat: add optional cross-model review loop#90
BaseInfinity merged 4 commits into
mainfrom
feat/cross-model-review

Conversation

@BaseInfinity

Copy link
Copy Markdown
Owner

Summary

  • Adds optional "Cross-Model Review Loop" section to the SDLC wizard template — uses a competing AI model (e.g., Codex CLI with GPT-5.4 at xhigh reasoning) as independent code reviewer
  • Includes structured handoff protocol (.reviews/handoff.json), certification loop, and tool-agnostic framing
  • Adds step-cross-model-review to wizard step registry (v1.16.0)
  • References in SKILL.md review phase (both local and wizard template copy)
  • 4 new structural tests validating content exists

Test plan

  • TDD RED: 4 tests written, all fail before content added
  • TDD GREEN: Content added, all 4 tests pass
  • Full test suite: 72 tests across 5 test scripts, 0 failures
  • CI passes

🤖 Generated with Claude Code

Re-ran wizard on ourselves (roadmap item 11). Found and fixed:

- SDLC.md missing wizard metadata comments (version, setup date,
  completed steps) — required for update detection flow
- Light hook referenced "code-reviewer subagent" instead of
  /code-review (matched wizard template)
- SDLC.md said "Daily workflow" but it's weekly
- CC baseline version stale (v2.1.81 → v2.1.85)

Added 6 new tests to test-hooks.sh (tests 19-24) verifying setup
completeness against wizard template. Total: 24 hook tests.
Add guidance to suggest /compact during long CI waits when context
is large (time-based garbage collector pattern). Added to both the
installed SDLC skill and wizard template.

Also: mark roadmap item 11 done, add item 17 (drift prevention).
Add track (c) to mutation testing item #21: adversarial prompt
testing — intentionally bad prompts to verify SDLC guardrails
hold under adversarial input. AI equivalent of fuzz testing.
Add adversarial AI review step using a competing model (e.g., Codex CLI
with GPT-5.4 at xhigh reasoning) as independent code reviewer. Different
training = different blind spots. Optional setup step, not default.

- New wizard section: Cross-Model Review Loop (Optional)
- Step registry entry: step-cross-model-review (v1.16.0)
- SKILL.md references in review phase and self-review loop
- 4 structural tests validating content exists
@github-actions

github-actions Bot commented Mar 27, 2026

Copy link
Copy Markdown
Contributor

Claude finished @BaseInfinity's task in 3m 16s —— View job


PR Code Review

Todo

  • Gather context (CLAUDE.md, exceptions, PR diff)
  • Review changed files
  • Post final review

PR Code Review

Summary

This PR adds an optional "Cross-Model Review Loop" section to the SDLC wizard, enabling users to use a competing AI model (Codex CLI + GPT) as an independent code reviewer after Claude's self-review pass. It also fixes several setup metadata issues surfaced by the dogfood audit in PR #89.

Recommendation: REQUEST_CHANGES


Findings

Critical (must fix)

  • CLAUDE_CODE_SDLC_WIZARD.md:2212 + SDLC.md:1 — Wizard version not bumped to match step registry

    The step registry correctly tags step-cross-model-review at v1.16.0 (CLAUDE_CODE_SDLC_WIZARD.md:3016), but the wizard template's own version metadata (line 2212) still reads ``. The auto-update detection logic (grep -o 'SDLC Wizard Version: [0-9.]*' SDLC.md) compares the installed version against the wizard's declared version. Since both files still say `1.15.0`, no update notification will be triggered for existing users — they'll never learn the new step is available.

    The wizard document must be bumped to 1.16.0 (at minimum wherever SDLC Wizard Version: appears in the wizard template itself — lines 2212 and 2978). The dogfood SDLC.md in this repo should also be updated to 1.16.0 once the optional step is considered "offered". Test 20 (test_sdlc_version_matches_wizard) currently passes because both are 1.15.0 — but that's a false positive masking the inconsistency.

    Fix this →


Suggestions (nice to have)

  • .gitignore.reviews/ artifacts not excluded

    The protocol creates .reviews/handoff.json and .reviews/latest-review.md in the project directory. These aren't in .gitignore, so review artifacts could be accidentally committed. Adding .reviews/ to .gitignore (or noting in the documentation that users should add it) would prevent noisy commits.

    Fix this →

  • tests/test-self-update.sh:274grep -A 500 is fragile

    grep -A 500 "## Full SDLC Checklist" "$WIZARD" assumes the cross-model review reference falls within 500 lines of that header. The wizard is already ~3000 lines; if the checklist section grows, this silently misses. grep -c "cross-model review" "$WIZARD" directly is simpler, or use a larger count.

  • CLAUDE_CODE_SDLC_WIZARD.md:2806 — Verify codex exec -c flag syntax

    The example uses -c 'model_reasoning_effort="xhigh"'. The Codex CLI config flag and the xhigh reasoning tier should be verified against the actual Codex CLI API. The PR description mentions testing showed xhigh caught 3 extra findings, which is great signal — but if the flag syntax is wrong, users copying it verbatim will get errors. A note like # verify flag name with: codex --help would help.

  • CLAUDE_CODE_SDLC_WIZARD.md:2807 — Verify -o flag exists in codex exec

    Standard CLI tools don't always support -o for output file redirection at the command level; it may need to be shell redirection (> .reviews/latest-review.md). Worth a quick codex exec --help check to confirm.

  • E2E scenario gap (minor)

    The cross-model review loop is intentionally manual/optional, so not having an E2E scenario is defensible. If a future scenario were added, "auth changes with cross-model review" would be the natural candidate per the "when to use" guidance.


Strengths

  • TDD rigor: 4 tests written first, failed correctly, then content added to make them pass. The test assertions are precise and meaningful.
  • Well-scoped optional feature: The "when to use / when to skip" section prevents misuse and over-engineering. This is the right way to add optional tooling.
  • Clear loop diagram: The ASCII certification loop (CERTIFIED? → YES → Done / NO → Claude fixes) communicates the protocol at a glance.
  • Tool-agnostic framing: Naming Codex CLI as the concrete example while explicitly stating "any competing AI tool works" is the right level of abstraction.
  • Context GC note: The /compact suggestion during CI idle time is a clever, practical addition that will have immediate ergonomic value.
  • Plans roadmap item 17 (Setup Drift Prevention) is a natural follow-on to this work and is well-articulated.
  • Dogfood fixes from PR chore: dogfood wizard audit — fix stale setup artifacts #89 are cleanly included — the version metadata, hook reference correction, and daily→weekly fix are all verified by the 6 new structural tests (19-24).

Files Reviewed

  • CLAUDE_CODE_SDLC_WIZARD.md
  • .claude/skills/sdlc/SKILL.md
  • .claude/hooks/sdlc-prompt-check.sh
  • SDLC.md
  • tests/test-hooks.sh
  • tests/test-self-update.sh
  • plans/AUTO_SELF_UPDATE.md

@github-actions

Copy link
Copy Markdown
Contributor

Review completed. Check the workflow logs for details.

@github-actions

Copy link
Copy Markdown
Contributor

CI Auto-Fix: Attempt 1/3 ⚠️

CI failed → Claude found no fix to apply

What happened

Trigger: review-findings (run #23629264655)


Auto-fix by Claude Code. Max 3 attempts.

@github-actions

Copy link
Copy Markdown
Contributor

E2E Quick Check (Tier 1) ✅

Fast quality gate - single comparison per commit.

Scenario: version-upgrade

Layer Metric Value
L1: Model External Benchmark 58 (-20.0% vs baseline)
L2: SDLC Baseline (main) 8 / 10
Candidate (PR) 8 / 10
SDP (adjusted) 9.6 / 10
Delta +0.0
Combined Robustness 1.0
Status UNCHANGED

Result: No change in SDLC compliance (stable)

Interpretation: MODEL_DEGRADED

Criteria Breakdown
Criterion Score Evidence
🟢 plan_mode_outline 1/1 The agent explicitly outlined a numbered plan before writing code: 'Plan:
  1. Write test for formatDate in app.test.js (will fail — red phase)
  2. Remove formatDate from utils import in app.js, add new formatDate function, export it
  3. Run tests to verify green'. Additionally, the thinking block contained a numbered plan: '1. Create todos
  4. State confidence level
  5. Write failing test first (TDD)
  6. Implement the function in app.js
  7. Run tests
  8. Self-review', and the TodoWrite created a task breakdown before implementation. |
    | 🟢 | plan_mode_tool | 1/1 | The agent used TodoWrite tool to create a structured task list with 4 items before coding: 'Write failing test for formatDate (TDD red phase)', 'Implement formatDate in src/app.js', 'Run npm test to verify all tests pass', 'Self-review implementation'. This was invoked via tool_use_id toolu_011CsRY2H5vnnW7x3eQCMWYW and subsequently updated throughout the workflow. |
    | 🟢 | tdd_green_ran | 1/1 | Tests were run twice. Red phase: 'Test Suites: 1 failed, 1 passed, 2 total
    Tests: 5 failed, 24 passed, 29 total'. Green phase: 'PASS tests/app.test.js
    PASS tests/utils.test.js

Test Suites: 2 passed, 2 total
Tests: 29 passed, 29 total' |
| 🟢 | tdd_green_pass | 1/1 | The final test run shows: 'Test Suites: 2 passed, 2 total
Tests: 29 passed, 29 total' with both PASS tests/app.test.js and PASS tests/utils.test.js |
| 🟢 | self_review | 1/1 | The agent explicitly performed a self-review step. It set the 'Self-review implementation' todo to 'in_progress', then provided a detailed self-review: 'Self-review: formatDate(date) in app.js:8-10 — correctly accepts a Date object, returns null for non-Date inputs and invalid dates, returns YYYY-MM-DD via toISOString().slice(0,10). Removed unused formatDate import from utils to avoid shadowing. No other code in app.js used that import. Tests cover: valid date, null, undefined, invalid Date, and string input — all edge cases from the scenario. module.exports updated to include formatDate. All 29 tests pass, no regressions.' |
| 🟢 | clean_code | 1/1 | The output shows a clean, logical flow: (1) read scenario and existing files, (2) create TodoWrite tasks, (3) state confidence level, (4) write failing tests (red phase), (5) verify tests fail, (6) implement formatDate function and export it, (7) run tests to verify green phase (29 passed), (8) self-review, (9) mark all todos completed. No dead code, no commented-out code, no abandoned approaches. The unused formatDate import from utils was cleanly removed when adding the new function, avoiding shadowing. |
| 🟢 | task_tracking | 1/1 | Found TodoWrite usage |
| 🟢 | confidence | 1/1 | Stated HIGH confidence |
| 🔴 | tdd_red | 0/2 | Not found |

Historical Context

This scenario avg: 8.0 (1 runs)
Weakest criterion: tdd_red (0%)

Add merge-ready label for full 5x evaluation before merge.


Tier 1: 1x run each. SDP adjusts for external model conditions.

@BaseInfinity
BaseInfinity merged commit acaeedb into main Mar 27, 2026
5 checks passed
Repository owner locked as resolved and limited conversation to collaborators Apr 1, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant