feat(#382): a crash mid-cycle no longer loses the work silently - #383
Merged
Conversation
`/work` is meant to be something you fire and walk away from. Until now a Pi process death mid-cycle lost the cycle AND left the state file asserting otherwise. The resume machinery was declared but inert: - `resumable: false` was a literal in the TYPE, so it could never be anything else. - `inFlightJobIds` was declared, validated and rendered — and never written anywhere in `src/`. - `dispatch-started` was never emitted at all, so the validator that cross-checks in-flight ids against it could only ever pass vacuously. - State was persisted only at step boundaries, while a single dispatch can run for thirty minutes. A crash inside that window left the file at the PREVIOUS boundary, still saying `status: "running"`. A dead cycle and a live one were indistinguishable, forever. This repo still carries the evidence: `.pi/work-state/547.json` and `551.json` sit at `running` with empty event logs. **Write-ahead.** Every step now persists a `dispatch-started` event, the in-flight job id and the owning pid BEFORE awaiting a dispatch. All nine steps, not just the five sharing `runSingleDispatch` — `explore`, `plan`, `develop` and `handoff` have their own dispatch shapes, and `develop` is the longest-running step in the cycle, so covering only the shared helper would have left the biggest crash window uncovered. **Ownership.** A `running` state file is either a live driver's or a corpse's, and those need opposite responses. Re-entry now resolves to: refuse (owner alive — two drivers on one branch interleave commits), resume (owner gone, dispatch was in flight), or continue (clean step boundary, as before). Resume granularity is the STEP. The child process died and its work with it, so the step starts over rather than continuing mid-flight. Sound because every step is dispatch-then-verify and the verify gates catch partial work; `commit-pr` and `merged` carry their own idempotency. An in-flight id with no matching `dispatch-started` cannot have come from the write-ahead — that is corrupt state, and the existing inconsistency halt still fires on it. The pre-existing skeleton test caught this distinction being collapsed. Orphaned `dispatch-started` events are kept: they are the only record that a dispatch was paid for and lost. **Queue state.** A multi-issue run now persists its outcome to `.pi/work-state/queue-summary.json` — parked reasons, human actions, and the groups that never started. Those leave no state file at all, so the run that parked them was the only place they were ever named, and it died with the session. `/work-status` with no issue argument now renders the multi-cycle index. `discoverAllCycles` and `renderCycleIndex` were written for exactly this and had NO caller anywhere in `src/`, so after a queue run the operator could only inspect one cycle at a time. `work-status.ts` crossed the §12 500-line cap; the multi-cycle view moved to `work-status-index.ts`. Tests: `test-resume.ts` proves the write-ahead reaches DISK by reading the state file back from inside the dispatch — a throw would not do, since `runSingleDispatch` catches those and the cycle continues. Also covers the three-way classification, the corrupt-vs-crashed distinction, and queue-summary round-tripping. `PI_ENSEMBLE_RESUME=0` makes six of those assertions fail, so the mechanism is load-bearing. Docs: README env table, a "Resume" and a "Queue state" section in docs/troubleshooting.md, AGENTS.md driver notes. Escape hatch: PI_ENSEMBLE_RESUME=0. Closes #382
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #382.
The problem
/workis meant to be something you fire and walk away from. Until now a Pi process death mid-cycle lost the cycle and left the state file asserting otherwise.The resume machinery was declared but inert:
resumable: falseworkflow-state-schema.ts:305) — could never be anything elseinFlightJobIds:399), rendered (work-status.ts:263) — never written anywhere insrc/dispatch-startedSo a crash inside a dispatch left the file at the previous boundary still saying
status: "running". A dead cycle and a live one were indistinguishable, forever. This repo still carries the evidence —.pi/work-state/547.jsonand551.jsonsit atrunningwith empty event logs.What this does
Write-ahead. Every step persists a
dispatch-startedevent, the in-flight job id and the owning pid before awaiting. All nine steps, not just the five sharingrunSingleDispatch—explore,plan,developandhandoffhave their own dispatch shapes, anddevelopis the longest-running step in the cycle, so covering only the shared helper would have left the biggest window uncovered.Ownership. A
runningstate file is either a live driver's or a corpse's, and those need opposite responses:Resume granularity is the step. The child process died and its work with it, so the step starts over rather than continuing mid-flight. That is sound because every step is dispatch-then-verify and the verify gates catch partial work;
commit-prandmergedadditionally carry their own idempotency (#362's PR pre-flight, already-merged tolerance).An in-flight id with no matching
dispatch-startedcannot have come from the write-ahead — that is corrupt state, and the existing inconsistency halt still fires on it rather than being silently cleared. The pre-existingtest-work-driver-skeleton.tscaught me collapsing that distinction. Orphaneddispatch-startedevents are deliberately kept: they are the only record that a dispatch was paid for and lost.Queue state. A multi-issue run now persists its outcome to
.pi/work-state/queue-summary.json— parked reasons, human actions, and the groups that never started. Those leave no state file at all, so the run that parked them was the only place they were ever named, and it died with the session.Dead code found and made live
discoverAllCyclesandrenderCycleIndex(work-status.ts) were written for the multi-cycle view and had no caller anywhere insrc/./work-statusonly ever reported on one issue, so after a multi-issue queue run the operator could inspect one cycle at a time and had no way at all to see groups that never started./work-statuswith no issue argument now renders the index; an explicit issue number still gives the detail view.Proving it works
test-resume.tsproves the write-ahead reaches disk by reading the state file back from inside the dispatch. A throw would not do —runSingleDispatchcatches those and the cycle continues, so the throw proves nothing about what is already persisted when a process dies. Also covers the three-way classification, the corrupt-vs-crashed distinction, and queue-summary round-tripping.PI_ENSEMBLE_RESUME=0makes six of those assertions fail, so the mechanism is load-bearing rather than decorative.Notes
work-status.tscrossed the §12 500-line cap; the multi-cycle view moved towork-status-index.ts.PI_ENSEMBLE_RESUME=0.docs/troubleshooting.md, AGENTS.md driver notes.Quality gate
bun run build✅ ·bunx tsc --noEmit✅ ·bun run check✅ · 77/77 offline smoke tests ✅