Skip to content

feat(#382): a crash mid-cycle no longer loses the work silently - #383

Merged
randomm merged 1 commit into
mainfrom
fix/issue-382-resume
Aug 7, 2026
Merged

feat(#382): a crash mid-cycle no longer loses the work silently#383
randomm merged 1 commit into
mainfrom
fix/issue-382-resume

Conversation

@randomm

@randomm randomm commented Aug 7, 2026

Copy link
Copy Markdown
Owner

Closes #382.

The problem

/work is meant to be something you fire and walk away from. Until now a Pi process death mid-cycle lost the cycle and left the state file asserting otherwise.

The resume machinery was declared but inert:

Thing Reality
resumable: false A literal in the type (workflow-state-schema.ts:305) — could never be anything else
inFlightJobIds Declared, validated (:399), rendered (work-status.ts:263) — never written anywhere in src/
dispatch-started Never emitted at all. The validator cross-checking it could only pass vacuously
Persistence Step boundaries only, while one dispatch can run 30 minutes

So a crash inside a dispatch left the file at the previous boundary still saying status: "running". A dead cycle and a live one were indistinguishable, forever. This repo still carries the evidence — .pi/work-state/547.json and 551.json sit at running with empty event logs.

What this does

Write-ahead. Every step persists a dispatch-started event, the in-flight job id and the owning pid before awaiting. All nine steps, not just the five sharing runSingleDispatchexplore, plan, develop and handoff have their own dispatch shapes, and develop is the longest-running step in the cycle, so covering only the shared helper would have left the biggest window uncovered.

Ownership. A running state file is either a live driver's or a corpse's, and those need opposite responses:

Situation Action
Recorded owner alive and not us Refuse — two drivers on one branch interleave commits and produce a PR nobody can review
Owner gone, dispatch was in flight Resume at the step that was in flight; completed steps are not re-dispatched
Nothing in flight Continue from the boundary, as before

Resume granularity is the step. The child process died and its work with it, so the step starts over rather than continuing mid-flight. That is sound because every step is dispatch-then-verify and the verify gates catch partial work; commit-pr and merged additionally carry their own idempotency (#362's PR pre-flight, already-merged tolerance).

An in-flight id with no matching dispatch-started cannot have come from the write-ahead — that is corrupt state, and the existing inconsistency halt still fires on it rather than being silently cleared. The pre-existing test-work-driver-skeleton.ts caught me collapsing that distinction. Orphaned dispatch-started events are deliberately kept: they are the only record that a dispatch was paid for and lost.

Queue state. A multi-issue run now persists its outcome to .pi/work-state/queue-summary.json — parked reasons, human actions, and the groups that never started. Those leave no state file at all, so the run that parked them was the only place they were ever named, and it died with the session.

Dead code found and made live

discoverAllCycles and renderCycleIndex (work-status.ts) were written for the multi-cycle view and had no caller anywhere in src/. /work-status only ever reported on one issue, so after a multi-issue queue run the operator could inspect one cycle at a time and had no way at all to see groups that never started. /work-status with no issue argument now renders the index; an explicit issue number still gives the detail view.

Proving it works

test-resume.ts proves the write-ahead reaches disk by reading the state file back from inside the dispatch. A throw would not do — runSingleDispatch catches those and the cycle continues, so the throw proves nothing about what is already persisted when a process dies. Also covers the three-way classification, the corrupt-vs-crashed distinction, and queue-summary round-tripping.

PI_ENSEMBLE_RESUME=0 makes six of those assertions fail, so the mechanism is load-bearing rather than decorative.

Notes

  • work-status.ts crossed the §12 500-line cap; the multi-cycle view moved to work-status-index.ts.
  • Escape hatch: PI_ENSEMBLE_RESUME=0.
  • Docs: README env table, "Resume" + "Queue state after you walk away" sections in docs/troubleshooting.md, AGENTS.md driver notes.

Quality gate

bun run build ✅ · bunx tsc --noEmit ✅ · bun run check ✅ · 77/77 offline smoke tests ✅

`/work` is meant to be something you fire and walk away from. Until now a
Pi process death mid-cycle lost the cycle AND left the state file
asserting otherwise. The resume machinery was declared but inert:

- `resumable: false` was a literal in the TYPE, so it could never be
  anything else.
- `inFlightJobIds` was declared, validated and rendered — and never
  written anywhere in `src/`.
- `dispatch-started` was never emitted at all, so the validator that
  cross-checks in-flight ids against it could only ever pass vacuously.
- State was persisted only at step boundaries, while a single dispatch
  can run for thirty minutes. A crash inside that window left the file at
  the PREVIOUS boundary, still saying `status: "running"`.

A dead cycle and a live one were indistinguishable, forever. This repo
still carries the evidence: `.pi/work-state/547.json` and `551.json` sit
at `running` with empty event logs.

**Write-ahead.** Every step now persists a `dispatch-started` event, the
in-flight job id and the owning pid BEFORE awaiting a dispatch. All nine
steps, not just the five sharing `runSingleDispatch` — `explore`, `plan`,
`develop` and `handoff` have their own dispatch shapes, and `develop` is
the longest-running step in the cycle, so covering only the shared helper
would have left the biggest crash window uncovered.

**Ownership.** A `running` state file is either a live driver's or a
corpse's, and those need opposite responses. Re-entry now resolves to:
refuse (owner alive — two drivers on one branch interleave commits),
resume (owner gone, dispatch was in flight), or continue (clean step
boundary, as before).

Resume granularity is the STEP. The child process died and its work with
it, so the step starts over rather than continuing mid-flight. Sound
because every step is dispatch-then-verify and the verify gates catch
partial work; `commit-pr` and `merged` carry their own idempotency.

An in-flight id with no matching `dispatch-started` cannot have come from
the write-ahead — that is corrupt state, and the existing inconsistency
halt still fires on it. The pre-existing skeleton test caught this
distinction being collapsed. Orphaned `dispatch-started` events are kept:
they are the only record that a dispatch was paid for and lost.

**Queue state.** A multi-issue run now persists its outcome to
`.pi/work-state/queue-summary.json` — parked reasons, human actions, and
the groups that never started. Those leave no state file at all, so the
run that parked them was the only place they were ever named, and it died
with the session.

`/work-status` with no issue argument now renders the multi-cycle index.
`discoverAllCycles` and `renderCycleIndex` were written for exactly this
and had NO caller anywhere in `src/`, so after a queue run the operator
could only inspect one cycle at a time.

`work-status.ts` crossed the §12 500-line cap; the multi-cycle view moved
to `work-status-index.ts`.

Tests: `test-resume.ts` proves the write-ahead reaches DISK by reading the
state file back from inside the dispatch — a throw would not do, since
`runSingleDispatch` catches those and the cycle continues. Also covers
the three-way classification, the corrupt-vs-crashed distinction, and
queue-summary round-tripping. `PI_ENSEMBLE_RESUME=0` makes six of those
assertions fail, so the mechanism is load-bearing.

Docs: README env table, a "Resume" and a "Queue state" section in
docs/troubleshooting.md, AGENTS.md driver notes.

Escape hatch: PI_ENSEMBLE_RESUME=0.

Closes #382
@randomm
randomm merged commit 1f5f97e into main Aug 7, 2026
1 check passed
@randomm
randomm deleted the fix/issue-382-resume branch August 7, 2026 07:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(work-driver): a crash mid-dispatch loses the cycle — resume is declared but never implemented

1 participant