roadmap(#212): local-max e2e shepherd foundation by BaseInfinity · Pull Request #229 · BaseInfinity/claude-sdlc-harness

BaseInfinity · 2026-04-24T00:44:41Z

Summary

Foundation for ROADMAP #212 — Local-Max E2E shepherd replacing CI's claude-code-action@v1 API-based simulation with a local claude --print run on Max subscription.

Flag OFF. CI unchanged. Zero behavior change for existing workflows. This PR ships only the shepherd script + tests + provenance schema. CI switchover is a follow-up after the Prove-It Gate runs.

What lands

tests/e2e/local-shepherd.sh — new orchestrator script
- Fork-PR abort via gh pr view --json headRepository.isFork (P0 trust boundary, Codex-hardened)
- Parity flags matching .github/workflows/ci.yml:327-358 byte-for-byte (--model claude-opus-4-7 --max-turns 55 --allowedTools ... --output-format json)
- Provenance fields on score-history: execution_path=local-max, host_os, cli_version, claude_code_version, auth_mode=subscription, pr_number
- GitHub check-run emission via gh api with dedicated e2e-local-shepherd name (does NOT collide with CI's e2e-quick-check)
- SDLC_LOCAL_SHEPHERD_DRY_RUN=1 for safe first-run inspection without posting
tests/test-local-shepherd.sh — 8 tests, all passing
- script-exists, usage-error, fork-abort (rc=2), missing-api-key-abort, parity-flags, provenance-fields, check-run-post, dry-run-skips-side-effects
- All fully mocked (no real claude, gh, curl calls)
ROADMAP Community Digest: Week of 2026-04-23 #212 row updated to FOUNDATION SHIPPED — FLAG OFF with remaining scope explicit

What's pending before CI switchover

Prove-It Gate: paired sims × 5 scenarios × 3 runs each (N=15/path) with 95% CI overlap
ci.yml migration to make local shepherd the default path, API fallback only for fork PRs
Docs update in CLAUDE_CODE_SDLC_WIZARD.md + skills/sdlc/SKILL.md

Codex findings addressed (from prior plan review)

Severity	Finding	Addressed by
P0	Trust boundary: local auth exposure to fork PRs	`gh pr view --json headRepository.isFork` abort (rc=2, distinct)
P1	"Zero-API" claim false (evaluator still hits API)	Scope honestly documented as partial-API; #228 tracks evaluator migration
P1	Sticky comments ≠ check-runs for branch protection	`gh api repos/.../check-runs` POST with dedicated check name
P1	Provenance required to avoid poisoning analytics	6 fields on every score-history entry
P1	Weak Prove-It gate (3 PRs × 95% CI too narrow)	Foundation flag OFF until N=15 paired sims verified

Test plan

bash tests/test-local-shepherd.sh — 8/8 green
bash tests/test-hooks.sh — 115 pass, 1 pre-existing unrelated fail
bash tests/test-compliance.sh — green
bash tests/test-doc-consistency.sh — 22/0 green
Codex cross-model review (xhigh) — running
CI validate green

Follow-ups scheduled

Next session: run Prove-It Gate locally, publish findings
After Gate: ci.yml migration PR
Lock live-session auto-setup as the default human path #228: evaluator migration off API

Foundation for ROADMAP #212 — Local-Max E2E shepherd replacing CI's claude-code-action@v1 API-based simulation with a local claude --print run on the maintainer's Max subscription. Scope shipped (foundation only, flag OFF, CI unchanged): - tests/e2e/local-shepherd.sh — new orchestrator script - tests/test-local-shepherd.sh — 8 tests covering: * script exists + executable * usage-error on no args * fork-PR abort (rc=2, P0 trust boundary) * missing ANTHROPIC_API_KEY abort (evaluator dep) * parity flags (--model claude-opus-4-7, --max-turns 55, --allowedTools ..., --output-format json) * provenance fields on score-history.jsonl (execution_path, host_os, cli_version, claude_code_version, auth_mode, pr_number) * GitHub check-run POST via gh api * dry-run mode skips side effects Parity matches .github/workflows/ci.yml:327-358 byte-for-byte. Evaluator still hits API (ROADMAP #228 will migrate). Pending before CI switchover: - Prove-It Gate: paired sims × 5 scenarios × 3 runs (N=15/path) - ci.yml migration with API as fork-PR fallback - docs update in SKILL.md + CLAUDE_CODE_SDLC_WIZARD.md Tests: 8/8 green locally. Adjacent suites unchanged (test-hooks 115/1, test-compliance pass, test-doc-consistency 22/0 — 1 pre-existing failure in test-hooks unrelated to this change). Addresses Codex P0/P1 findings from plan review in .reviews/ latest-review-pr223.md: - P0 trust boundary: fork PR gh-query abort (exit 2, distinct) - P1 zero-API claim: honestly documented as partial-API - P1 check-run emission: gh api repos/.../check-runs - P1 provenance fields: 6 fields per score-history entry - P1 parity gate: foundation behind flag, not default

test-local-shepherd.sh needs to run in CI validate alongside other test-*.sh scripts. Also added to CONTRIBUTING.md two-list registry (both the oneliner and the enumerated list). Caught by tests/test-workflow-triggers.sh which asserts all test-*.sh files are referenced in ci.yml + CONTRIBUTING.md.

…rity Codex cross-model review of PR #229 scored 3/10 NOT CERTIFIED with 3 real catches. All addressed: LS-001 (P0) SHA verification missing Before: shepherd posted a check-run on PR head SHA while scoring whatever local state happened to be checked out — could certify wrong code as passing. After: `git rev-parse HEAD` must match `gh pr view --json headRefOid` before scoring, history append, or check-run POST. Clear error tells user to `gh pr checkout <PR>`. Override via SDLC_SHEPHERD_SKIP_SHA_CHECK=1. LS-002 (P0) Exit-code contract broken Before: `claude ... || true` swallowed sim failures; evaluator errors became score=0/10 and were indistinguishable from real failing simulations; all paths returned rc=0. After: claude non-zero propagates rc=1; evaluator non-zero propagates rc=1; nothing appended to score-history on failure. rc=2 is reserved for fork-PR abort only. SDLC_SHEPHERD_SOFT_FAIL=1 restores lenient mode for smoke testing. LS-003 (P1) Prompt/flag parity not byte-equal Before: shepherd passed raw scenario task text; missing --add-dir; no prompt preamble. After: full CI prompt embedded verbatim (the 7-STEPS + IMPORTANT block from .github/workflows/ci.yml:338-361), --add-dir added, plus a parity-diff test that greps both files for signature lines so divergence is caught at test-time rather than discovered via score drift weeks later. Also fixed a bash 3.2 on macOS heredoc-in-$() parser bug by rewriting the prompt construction as printf lines instead of <<PROMPT. Tests: 4 new precompact→shepherd tests (sha-mismatch, claude-crash-rc=1, evaluator-crash-rc=1, prompt-parity-diff). 12/12 shepherd tests green. Adjacent: test-hooks 115/1 (1 pre-existing), test-workflow-triggers clean. Codex verified: - Fork PR abort fail-closed (preflight item a) ✓ - No downstream breakage from provenance fields ✓ - --allowedTools comma-separated accepted by current CLI ✓ - e2e-local-shepherd check name non-colliding ✓

LS-002 residual: check-run POST failure was still rc=0 with a warning — branch protection would never know the gate was unsatisfied. Now rc=1 on check-run error (SDLC_SHEPHERD_SOFT_FAIL=1 restores warning mode). Added test_shepherd_exits_1_on_checkrun_failure asserting rc=1 when `gh api` POST fails. LS-003 residual: shepherd had --model claude-opus-4-7 but CI does not pin model. Removed the explicit pin — both paths rely on action/CLI default now; if Anthropic changes the default, both shift together. Updated parity test to assert --add-dir instead of --model. Codex notes-for-next-review suggested replacing the grep-based parity test with a full byte-diff; deferring to a follow-up PR since the current grep covers the CI lines that the original certify condition called out. Tests: 13/13 green (12 previous + 1 new check-run-rc=1). Codex round 2 score: 5/10 (NOT CERTIFIED). These fixes target the two remaining open issues.

* docs(handoff): document pr_number self-heal opt-in (#209, v1.42.2) ROADMAP #209 closure. The precompact-seam-check.sh hook self-heals on PENDING_* handoffs whose linked PR has merged: when handoff has pr_number and `gh pr view N --json state` returns MERGED, hook treats handoff as implicit CERTIFIED and unblocks /compact silently. Self-heal logic shipped earlier alongside #229 (stale-expire fallback) but was undocumented in the handoff template schemas, leaving the path effectively dead code on fresh installs (consumers had no way to discover the opt-in). Documented pr_number in all 3 handoff templates: - skills/sdlc/SKILL.md (Step 1: Mission-First Handoff) + key-fields prose - CLAUDE_CODE_SDLC_WIZARD.md (Round 1: Initial Review) - CLAUDE_CODE_SDLC_WIZARD.md (cross-model review section) New regression test test_handoff_template_documents_pr_number in tests/test-hooks.sh enforces template/doc parity going forward — a future schema edit that drops pr_number will fail this test. Together with #229 (mtime auto-expire) closes the "stuck PENDING handoff blocks /compact forever" footgun from both directions: PR-linked reviews self-heal on merge (instant), unlinked reviews auto-expire on mtime (14d default). Hit live in this repo 2026-04-19 (PR #205) and 2026-04-26 (PR #253) where handoffs lacked pr_number and fell through to the stale-expire fallback. Tests: 130 hook tests passing (1 new), 10 adjacent test files green. * fix(handoff-test): round-2 — coverage + prose + count (#255) Codex round-1 review (6/10 NOT CERTIFIED) found 3 issues: 1. P1: test_handoff_template_documents_pr_number false-greened on single-schema deletion. Required ANY occurrence per file, but wizard has 2 schemas — removing pr_number from one left the other intact and test passed. Fix: count handoff schemas via `"status": "PENDING_REVIEW"` marker (response.json uses PENDING_RECHECK so no collision). Assert pr_number occurrence count >= handoff count per file. Mutation re-verified: deleting pr_number from any of the 3 schemas now flips test to FAIL with skill-coverage(0/1) or wizard-coverage(1/2). Discovered separate grep -c quirk: `|| echo 0` after grep -c with no matches produces '0\n0' (grep -c always prints a number AND exits 1; || appends another 0). Switched to `|| true` + `${var:-0}` defaults — preserves count, discards failing exit, set -e safe. 2. P2: cross-model schema had pr_number field but no nearby prose explanation. Round 1 schema's inline explanation wasn't mirrored. Fix: added paragraph at CLAUDE_CODE_SDLC_WIZARD.md:3709 explaining when to set pr_number, what precompact-seam-check.sh does with it, when to omit. 3. P2: CHANGELOG overstated hook test count (130 vs actual 129). Fix: corrected to "129 hook tests, 1 new". All 3 mutation paths verified catching: - skill schema delete → FAIL skill-coverage(0/1) - wizard cross-model delete → FAIL wizard-coverage(1/2) - wizard Round 1 delete → FAIL wizard-coverage(1/2) Adjacent regression: 10 test files green (test-hooks 129/0 + 9 others).

Live-fire bug 2026-04-23: at session end, /compact was blocked by the precompact-seam hook because .reviews/handoff.json is stuck at PENDING_RECHECK from the review that SHIPPED the hook itself. That original review was written BEFORE the self-heal pr_number feature (#209) existed, so there's no PR to auto-query. Handoff is stuck forever until manually flipped. Every wizard user who did a seam review before #209 has this landmine in their repo. Fix: add stale-handoff auto-expire. PENDING_* + no pr_number + mtime > N days → treat as CERTIFIED with WARN (not HOLD). Rescues stale handoffs without bypassing live reviews. P1 because it silently blocks a core UX (manual /compact) and the user hit it mid-session.

Fix UX landmine where handoff.json reviews predating the #209 pr_number self-heal feature would block manual /compact forever. Hook now heals via two paths: - (a) #209: pr_number + gh says MERGED → silent unblock - (b) #229: no pr_number + mtime > SDLC_HANDOFF_STALE_DAYS (default 14) → unblock with one-line WARN PR-linked handoffs still ignore mtime — an OPEN PR with old mtime is still a live review. Tests: 4 new in test-hooks.sh covering stale-unblocks, fresh-still-blocks, stale-with-pr_number-prefers-self-heal, threshold-override. 19/19 precompact tests green. Caught live-fire 2026-04-23 when session-end /compact blocked on a review that shipped the hook itself.

CI caught a real portability bug: `stat -f %m` on Linux is `--file-system` (dumps FS info to stdout, exit 0), not format. My `||` fallback never fired, $MTIME got garbage, and arithmetic blew up on every PENDING handoff path — regressing the existing PENDING_REVIEW/RECHECK blocks. Fix: try GNU `stat -c %Y` first (BSD errors → falls through). Add numeric case-guard so any junk in $MTIME skips the stale check entirely and falls through to HOLD (safe default). Locally macOS green. Pushing to verify Linux CI passes now.

Codex review of PR #227 caught: `SDLC_HANDOFF_STALE_DAYS=foo` leaked `[: foo: integer expression expected` to stderr before the normal HOLD text. A typo in the env var should silently fall back, not emit shell diagnostics. Fix: non-negative-integer case-pattern guard. Empty, non-numeric, negative, or decimal values fall back to default 14. Test: test_precompact_stale_threshold_invalid_falls_back asserts rc=2, PENDING_REVIEW in stderr, and NO `integer expression` noise. 20/20 precompact tests green. Codex score: 8/10 CERTIFIED (this addresses the sole P2 finding).

…rity Codex cross-model review of PR #229 scored 3/10 NOT CERTIFIED with 3 real catches. All addressed: LS-001 (P0) SHA verification missing Before: shepherd posted a check-run on PR head SHA while scoring whatever local state happened to be checked out — could certify wrong code as passing. After: `git rev-parse HEAD` must match `gh pr view --json headRefOid` before scoring, history append, or check-run POST. Clear error tells user to `gh pr checkout <PR>`. Override via SDLC_SHEPHERD_SKIP_SHA_CHECK=1. LS-002 (P0) Exit-code contract broken Before: `claude ... || true` swallowed sim failures; evaluator errors became score=0/10 and were indistinguishable from real failing simulations; all paths returned rc=0. After: claude non-zero propagates rc=1; evaluator non-zero propagates rc=1; nothing appended to score-history on failure. rc=2 is reserved for fork-PR abort only. SDLC_SHEPHERD_SOFT_FAIL=1 restores lenient mode for smoke testing. LS-003 (P1) Prompt/flag parity not byte-equal Before: shepherd passed raw scenario task text; missing --add-dir; no prompt preamble. After: full CI prompt embedded verbatim (the 7-STEPS + IMPORTANT block from .github/workflows/ci.yml:338-361), --add-dir added, plus a parity-diff test that greps both files for signature lines so divergence is caught at test-time rather than discovered via score drift weeks later. Also fixed a bash 3.2 on macOS heredoc-in-$() parser bug by rewriting the prompt construction as printf lines instead of <<PROMPT. Tests: 4 new precompact→shepherd tests (sha-mismatch, claude-crash-rc=1, evaluator-crash-rc=1, prompt-parity-diff). 12/12 shepherd tests green. Adjacent: test-hooks 115/1 (1 pre-existing), test-workflow-triggers clean. Codex verified: - Fork PR abort fail-closed (preflight item a) ✓ - No downstream breakage from provenance fields ✓ - --allowedTools comma-separated accepted by current CLI ✓ - e2e-local-shepherd check name non-colliding ✓

Kill the Anthropic API burn in CI by removing the e2e simulation jobs entirely. Shepherd (shipped in #229) stays as the local advisory tool. Rationale (per Codex architect review .reviews/architect-proposal.md): Original plan (pending check-run + local overwrite) got 2/10 NOT CERTIFIED because branch protection is pinned to GitHub Actions app_id:15368 — a local gh OAuth token cannot satisfy a required check. Option 1 matches reality (we admin-merge around red e2e every time anyway) and cleanly unblocks fork PRs. Changes: - ci.yml: remove e2e-quick-check + e2e-full-evaluation jobs (1534 lines deleted). Keep validate + cleanup-old-comments. - weekly-update.yml: disable cron (9 claude-code-action blocks still present, tracked for removal in ROADMAP #231). Manual via workflow_dispatch only. - monthly-research.yml: same — disable cron. - pr-review.yml: wait on validate only (e2e check no longer exists). - Tests: stub obsolete test-degradation + test-workflow-triggers tests that asserted deleted ci.yml structure. Shepherd parity test reworked to assert signatures exist in shepherd (ci.yml parity is moot now). - ROADMAP: #212 updated to "OPTION 1 SHIPPING"; added #230 (shepherd baseline/candidate extension) + #231 (weekly/monthly workflow cleanup). What this PR does NOT change (intentional, follow-up items): - Shepherd stays candidate-only (no baseline comparison) — ROADMAP #230 - Weekly/monthly workflows still have claude-code-action blocks — ROADMAP #231 - Branch protection is NOT patched in this PR — that PATCH happens separately via `gh api` just before admin-merge to avoid a window where new PRs have no e2e check producer but still require it Signal loss accepted: CI no longer auto-scores PRs. Shepherd must be run manually on each PR the maintainer wants to score. Matches today- reality of admin-merge-every-time. Tests passing locally: - test-workflow-triggers: 54/0 - test-local-shepherd: 13/0 - test-hooks: 115/1 (pre-existing failure) - test-degradation-detection: 14/0 - test-cusum: pass - test-doc-consistency: 22/0 Branch protection will be patched separately via: gh api repos/.../branches/main/protection/required_status_checks \\ --method PATCH --field checks=[{context=validate,app_id=15368}]

P1 UX landmine fix: precompact-seam-check blocked manual /compact when handoff.json was PENDING_* from a pre-#209 review with no pr_number. Now auto-expires handoffs older than SDLC_HANDOFF_STALE_DAYS (default 14) with a one-line WARN; PR-linked handoffs still use #209 self-heal path. Tests: 5 new precompact tests (stale-unblocks, fresh-still-blocks, stale-with-pr prefers-self-heal, threshold-override, invalid-env-falls-back). 20/20 precompact tests green. Codex cross-model review: 8/10 CERTIFIED. Sole P2 (bad env var leaking integer-expression error) fixed in round 2. Admin-merge justification: validate + Codex both green, real bar met. e2e-quick-check red is known API-credit-cap issue (same as PR #222); the API canary is intentionally dead to detect CI leaks. Real fix is ROADMAP #212 (local-Max shepherd, kills API dependency). Not admin-bypassing code review — admin-bypassing a billing infrastructure gate.

* docs(handoff): document pr_number self-heal opt-in (#209, v1.42.2) ROADMAP #209 closure. The precompact-seam-check.sh hook self-heals on PENDING_* handoffs whose linked PR has merged: when handoff has pr_number and `gh pr view N --json state` returns MERGED, hook treats handoff as implicit CERTIFIED and unblocks /compact silently. Self-heal logic shipped earlier alongside #229 (stale-expire fallback) but was undocumented in the handoff template schemas, leaving the path effectively dead code on fresh installs (consumers had no way to discover the opt-in). Documented pr_number in all 3 handoff templates: - skills/sdlc/SKILL.md (Step 1: Mission-First Handoff) + key-fields prose - CLAUDE_CODE_SDLC_WIZARD.md (Round 1: Initial Review) - CLAUDE_CODE_SDLC_WIZARD.md (cross-model review section) New regression test test_handoff_template_documents_pr_number in tests/test-hooks.sh enforces template/doc parity going forward — a future schema edit that drops pr_number will fail this test. Together with #229 (mtime auto-expire) closes the "stuck PENDING handoff blocks /compact forever" footgun from both directions: PR-linked reviews self-heal on merge (instant), unlinked reviews auto-expire on mtime (14d default). Hit live in this repo 2026-04-19 (PR #205) and 2026-04-26 (PR #253) where handoffs lacked pr_number and fell through to the stale-expire fallback. Tests: 130 hook tests passing (1 new), 10 adjacent test files green. * fix(handoff-test): round-2 — coverage + prose + count (#255) Codex round-1 review (6/10 NOT CERTIFIED) found 3 issues: 1. P1: test_handoff_template_documents_pr_number false-greened on single-schema deletion. Required ANY occurrence per file, but wizard has 2 schemas — removing pr_number from one left the other intact and test passed. Fix: count handoff schemas via `"status": "PENDING_REVIEW"` marker (response.json uses PENDING_RECHECK so no collision). Assert pr_number occurrence count >= handoff count per file. Mutation re-verified: deleting pr_number from any of the 3 schemas now flips test to FAIL with skill-coverage(0/1) or wizard-coverage(1/2). Discovered separate grep -c quirk: `|| echo 0` after grep -c with no matches produces '0\n0' (grep -c always prints a number AND exits 1; || appends another 0). Switched to `|| true` + `${var:-0}` defaults — preserves count, discards failing exit, set -e safe. 2. P2: cross-model schema had pr_number field but no nearby prose explanation. Round 1 schema's inline explanation wasn't mirrored. Fix: added paragraph at CLAUDE_CODE_SDLC_WIZARD.md:3709 explaining when to set pr_number, what precompact-seam-check.sh does with it, when to omit. 3. P2: CHANGELOG overstated hook test count (130 vs actual 129). Fix: corrected to "129 hook tests, 1 new". All 3 mutation paths verified catching: - skill schema delete → FAIL skill-coverage(0/1) - wizard cross-model delete → FAIL wizard-coverage(1/2) - wizard Round 1 delete → FAIL wizard-coverage(1/2) Adjacent regression: 10 test files green (test-hooks 129/0 + 9 others).

BaseInfinity added 4 commits April 23, 2026 17:44

BaseInfinity merged commit c100314 into main Apr 24, 2026
4 of 5 checks passed

This was referenced Apr 24, 2026

roadmap(#212): Option 1 — de-require e2e, shepherd becomes advisory #231

Merged

docs(handoff): pr_number self-heal opt-in (#209, v1.42.2) #255

Merged

BaseInfinity mentioned this pull request Apr 27, 2026

[enhancement] PreCompact hook should auto-CERTIFY when fixes_applied SHAs are already in HEAD's ancestry #257

Closed

Navigation Menu

Uh oh!

roadmap(#212): local-max e2e shepherd foundation - #229

roadmap(#212): local-max e2e shepherd foundation#229
BaseInfinity merged 4 commits into
mainfrom
roadmap-212-local-shepherd

BaseInfinity commented Apr 24, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

BaseInfinity commented Apr 24, 2026

Summary

What lands

What's pending before CI switchover

Codex findings addressed (from prior plan review)

Test plan

Follow-ups scheduled

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant