feat(work-driver): driver-side outcome-verification gates (executed evidence, not transcripts) - #270
Merged
Merged
Conversation
…vidence, not transcripts) Every quality gate before this PR was LLM judgment — adversarial + six lenses reading diffs and transcripts. Nothing driver-side ever EXECUTED anything until post-PR CI. Agents claim "done" and the driver trusted the claim; the #245/#253 silent-merge incidents were exactly this failure class (MAST: verification failures = 21.3% of multi-agent failures). Top-ranked item from the harness gap analysis — unanimous across operator-pain, competitive-gap, and reliability lenses. New machinery (all pure driver code, zero LLM tokens): - `verifyStepOutcome(ctx, state, step)` — the gate. - develop (runs when every branch claims success): at least one worktree has a real diff (porcelain, or commits ahead of baseSha); the project's verify command exits 0 in each changed worktree. - commit-pr (runs when the consolidation gate passes): commits exist ahead of origin/<base>; the parsed PR number resolves via `gh pr view`. Missing `pr:` marker triggers a `gh pr list --head` repair that ADOPTS the found number into pipelineState (pre-PR17 a missing marker silently degraded handoff/CI targeting). - `verifyCmdFor(repoRoot)` — verify-command discovery: `.pi/verify-cmd` file > package.json typecheck/test script with lockfile-detected runner (bun/pnpm/yarn/npm) > Cargo.toml → `cargo check --quiet` > none (diff-evidence-only mode, noted). - `PipelineState.baseSha` — recorded at branch step (git rev-parse HEAD right after branch creation); optional, no schema bump. - `PipelineState.verifyEvidence` — per-check failure evidence, rendered into the handoff body by explainCap. - New cap shape `verify-failed:<step>` → handoff. - `DriverContext.verifyExecFn` — test injection point (mirrors issueBodyFetcherFn). Env knobs: PI_ENSEMBLE_VERIFY=0 escape hatch; PI_ENSEMBLE_VERIFY_TIMEOUT_MS caps the verify command (default 10 min). Tests: 21 new assertions (discovery precedence, hollow-claim detection, verify-cmd failure evidence, PR adoption repair, explainCap rendering). The gate is disabled globally in test-work-driver.ts flow tests (fake tmp dirs aren't git repos); dedicated gate tests re-enable it with an injected executor. Verified: tsc + biome + all 26 offline smokes pass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Top-ranked item from the harness gap analysis (9-agent research workflow: codebase inventory + surveys of Claude Code/Codex/OpenHands/Aider + MAST failure-mode research, synthesized across operator-pain / competitive-gap / reliability lenses — unanimous #1 across all three).
The gap. Every quality gate before this PR is LLM judgment — adversarial + six lenses reading diffs and transcripts. Nothing driver-side ever executes anything until post-PR CI. Agents claim "done" and the driver trusts the claim; the #245/#253 silent-merge incidents were exactly this class (MAST: verification failures = 21.3% of multi-agent failures). Codex, OpenHands, and Claude Code hooks all gate on execution exit codes.
The fix. After
develop(all branches claiming success) and aftercommit-pr(consolidation gate passed), the driver checks executed evidence — zero LLM tokens:developbaseSharecorded at branch step); project verify command exits 0 in each changed worktreecommit-prorigin/<base>; PR number resolves viagh pr view; missingpr:marker triggersgh pr list --headrepair that adopts the found numberFailure → cap
verify-failed:<step>→ handoff, evidence inpipelineState.verifyEvidence, rendered byexplainCap.Verify-command discovery (
verifyCmdFor):.pi/verify-cmdfile > package.jsontypecheck/testscript with lockfile-detected runner (bun/pnpm/yarn/npm) >Cargo.toml→cargo check --quiet> none (diff-evidence-only, noted).Env knobs:
PI_ENSEMBLE_VERIFY=0escape hatch;PI_ENSEMBLE_VERIFY_TIMEOUT_MS(default 10 min).Changes
extension/src/work-driver.tsverifyStepOutcome+verifyCmdFor+ wiring in runDevelop/runCommitPr + baseSha capture in runBranch + explainCap branch +DriverContext.verifyExecFninjection pointextension/src/workflow-state.tsverify-failed:<step>cap shape;baseSha,verifyEvidenceoptional fields (no schema bump)extension/smoke-tests/test-work-driver.tsREADME.md,AGENTS.md,docs/troubleshooting.mdTest plan
bunx tsc --noEmitcleanbun run check(biome) clean/work Npasses the gates transparently (cargo check ~1-3 min per develop)verify-failed:develophandoff with evidencepr:marker repair: cycle where ops omits the marker → gate adopts the number viagh pr list --headNon-goals (next items from the gap analysis)
REPLAY_ONCEbudgeted replay before handoff) — separate PR; this PR's evidence attachment is a prerequisite.ALREADY_COMPLETEevidence verification — deferred to the follow-up.References
indexed-swimming-lovelace.md)