ref(chat): Simplify durable conversation execution - #1344
Conversation
Replace the five near-identical persist* session helpers with saveTurnCheckpoint(mode: running | paused | completed | failed). Resume uses that API only, adopts the committed boundary on continue, and fails closed when a slice parks again at the same resumed boundary. Append history checks durable message identity so Pi in-place envelope mutations no longer look like branches. SQL event history is unchanged; Redis remains a thin resume cursor. Co-Authored-By: David Cramer <david@sentry.io>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Resume now imports saveTurnCheckpoint; the sandbox component mock still only exported the old persist* helpers and failed the suite.
There was a problem hiding this comment.
This is a useful step toward the durable-queue model, but it does not yet land the containment you described for that system. Durable identity + a single write function are real improvements; the dual execution machines, old session-record surface, and compatibility layer remain the main problem.
Please treat this as a containment cut, not a rename of persist*:
- Contain the durable queue path. The PR still leaves conversation work (
task-execution) and turn-session resume as two machines that both answer “is this alive / can we continue?”. File names and public API still sayturn-session-record/SessionRecord/sessionId, while callers also sayturnId. For the core queue, a maintainer should be able to open one small module family and answer: mailbox, lease, checkpoint cursor, worker loop. - Hard-cut the old write API. Production still imports the five
persist*/completeDeliveredTurnshims. Policy requires either a hard cutover orTODO(vX.Y.Z)on compatibility paths. You already approved abandoning in-flight turns, so cut the callers and delete the shims in this PR (or the immediate follow-up before merge). - Shrink the public write shape.
TurnCheckpointBaseis still a bag of routing, metrics, provenance, and delivery fields. Keep the public API to the few fields the lifecycle needs; pass SQL dual-write/live routing through a narrower internal path if needed. - Use one vocabulary. New code should use
turnId(terminology), one pause/status word (pausedvs storedawaiting_resumeneeds one mapping, not two public dialects), and stop introducingSessionRecord/loadTurnSessionRecordnames next tosaveTurnCheckpoint.
The durable-identity change and resume baseline are the right local fixes for JUNIOR-62. I would not merge until the surface is small enough that “checkpoint” is obvious without reading eight files.
Delete the dual turn-session-record surface and one-hop session wrapper. One API lives with the durable queue: loadTurnCheckpoint / saveTurnCheckpoint. Callers and tests use turnId; shims are gone. SQL history untouched. Co-Authored-By: David Cramer <david@sentry.io>
There was a problem hiding this comment.
The hard cut to task-execution/checkpoint addresses the earlier API/ownership feedback. There are still two things to fix before this lands:
- The no-progress path needs behavior coverage that resumes an
awaiting_resumecheckpoint and proves an unchanged timeout/retry boundary becomes terminal. The current resume test only covers auth persistence, so the main fix for JUNIOR-62 is untested. - CI is currently red for PR-owned changes: the renamed checkpoint test exceeds the file-length limit while the exception still names the deleted file, and the cooperative-yield assertion expects the old error text. Please fix both rather than merging with the required check failing.
Runtime status is paused. Checkpoint + continue live under task-execution. SQL free-text/enums keep awaiting_resume via edge mapping only. No SQL schema change. In-flight resume records may drop on deploy. Co-Authored-By: David Cramer <david@sentry.io>
Only treat a running checkpoint as progress when the persisted boundary differs from the resumed one, and cover timeout plus same-boundary running writes in the no-progress regression.
Queue continue already holds the conversation work lease. Skip the extra thread lock and the ResumeTurnBusy retry loop so continue is one owner, not two. OAuth out-of-band resumes still lock. Continue reads go through loadTurnCheckpoint. Co-Authored-By: David Cramer <david@sentry.io>
Move continue-run and turn-cursor storage under the durable queue folder. Checkpoint is the only external gate; outside callers no longer import turn storage or the old runtime continue runner path. Co-Authored-By: David Cramer <david@sentry.io>
There was a problem hiding this comment.
Adversarial pass on complexity / terminology / dual surfaces. The hard-cut and no-progress tests moved the right direction, but this still reads like a rename of the old turn-session machine more than a thinner execution model.
Please fix these before merge:
- Containment is fake.
checkpoint.tsclaims to be the only external gate, then re-exportsgetTurnRecord/failTurnRecord/ summaries, and production still importsturn-cursordirectly (oauth-callback,reply-executor,agent-dispatch/work,agent-invocations/work,continue-run). One public module. Internal storage stays private. - One turn identity. Runtime still juggles
turnIdargs againstsessionIdfields /sessionRecordlocals /app.ai.resume_session_id.TurnRecord.sessionIdshould beturnIdat the TypeScript boundary; keep the Redis key mapping in one private place. turn-cursor.tsis still the old bag (~1100 LOC + exception). A "thin cursor" should not own SQL dual-write, runtime-context splice, history-version follow, and recovery index TTL policy in one file. Shrink or split by real concern — do not land another rename with a file-length exception.- Yield is outside the no-progress fail-closed path.
translateSuspensiononly checks same-boundary spin for timeout/retry. Cooperative yield can re-park the identical boundary forever whileshouldYield()stays true. Apply the same fail-closed rule (or an explicit slice/progress counter) to yield. - Checkpoint write shape is still a grab-bag.
TurnCheckpointWrite+sharedWriteoptional-merge every routing/metrics/provenance field. Public write should be lifecycle-only; SQL dual-write metadata should not dominate the API.
I would not ship this as "simplified conversation execution" until a maintainer can open one small module family and not also keep the old session vocabulary in their head.
| getTurnRecord, | ||
| listTurnSummaries, | ||
| recordTurnSummary, | ||
| }; |
There was a problem hiding this comment.
This re-export list breaks the claim two lines above that turn-cursor.ts is internal. Callers already import storage helpers from here and from ./turn-cursor directly. Delete these re-exports and force every production caller through the real checkpoint lifecycle API (load / save / maybe explicit fail+abandon wrappers that take turnId). If a read helper is truly needed outside this folder, give it a checkpoint-shaped name and stop exporting the storage record API.
There was a problem hiding this comment.
Addressed. Production code outside task-execution no longer imports turn-cursor. checkpoint.ts is the external gate. paused-turn.ts is part of the same execution owner and uses the private store directly.
| turnContexts?: PluginTurnContext[]; | ||
| durationMs?: number; | ||
| usage?: AgentTurnUsage; | ||
| errorMessage?: string; |
There was a problem hiding this comment.
TurnCheckpointWrite is still the old optional wall (destination/source/actor/skills/provenance/usage/dispatch receipts). That is not a lifecycle API. Split to required progress fields only; pass dual-write routing through a narrow internal path if SQL still needs it this PR.
| ).message, | ||
| sliceId: args.sliceId, | ||
| state: "failed", | ||
| }); |
There was a problem hiding this comment.
savePaused can return a record with state: "failed" on the slice-limit path. Callers then branch on record.state === "paused" vs throw. Prefer throwing TurnSliceLimitExceededError here (or returning a discriminated result) so "saved pause" cannot mean "terminal failure".
Co-Authored-By: David Cramer <david@sentry.io>
Co-Authored-By: David Cramer <david@sentry.io>
There was a problem hiding this comment.
I can follow the local mechanics, but I still cannot state the intended contract confidently enough to maintain this.
My current reading is: the conversation lease owns normal liveness; a checkpoint points into SQL history; paused timeout/retry/yield work is resumed by the worker; repeating a timeout/retry boundary fails the turn; a running checkpoint with no other owner is instead recovered; OAuth remains an out-of-band exception.
Please clarify which of these is the actual goal:
- Is this a narrow fix for JUNIOR-62, or a migration to a new durable conversation execution model? The title/body say “simplify,” but the diff also defines worker-death recovery, history-replacement following, legacy-state normalization, summary-index retention, and OAuth exceptions across 72 files.
- After a deploy or hard worker death, is preserving an in-flight
runningturn a required contract, or may it fail closed? Earlier PR text explicitly allowed in-flight turns to fail; current code adds recovery, while the current body no longer states the policy. This decision changes whether most ofrecoverStrandedRunningSessionand its fallbacks belong here. - Why is cooperative yield allowed to repeat the same boundary when timeout/retry are not? If yield is expected to make no model progress while handing off a lease, say that and identify what guarantees it eventually stops; otherwise it belongs under the same no-progress rule.
Please put the chosen state machine and failure policy in task-execution/README.md in plain terms: owner, legal states/transitions, what survives process death, and which exceptions are intentional. Right now those answers have to be inferred from several files and old review history.
There was a problem hiding this comment.
This still does not meet the stated one-queue/one-owner design, and two retry paths remain unsafe. Please address these before merge:
durable-queue.test.tsdoes not exercise the production continue path for JUNIOR-62. Its incident case injects a custom worker callback and manually callscreateResumeState; it bypassesresumeAwaitingSlackContinuation/continueSlackAgentRun,resumeSlackTurn, production routing, and the worker's actual paused-turn selection. It proves the local no-progress helper, not queue → production worker → checkpoint → redelivery. Keep the small harness, but make the incident scenario invoke the real worker entry with only the agent/external boundary injected.- Cooperative yield bypasses the no-progress guard in
agent/resume.ts. A resumed run that immediately yields at the same boundary can persistpausedforever because yield neither advances the slice nor compares againstresumedBoundaryKey. Apply the same fail-closed progress rule to yield and cover it in the high-level suite. savePausedcan return a terminalfailedrecord when the slice limit is reached.parkForAuthtreats every truthy result as a successful auth pause and returnsawaiting_auth, so a failed turn can be exposed as resumable. Make a pause operation either return a paused checkpoint or throw/return an explicit terminal result; callers must not infer success from truthiness.checkpoint.tsis not a small lifecycle boundary.TurnCheckpointWritestill exposes routing, actor, provenance, metrics, dispatch, and delivery fields, and the module re-exports raw record/summary storage operations. Seventeen production files import this surface, whilecontinue-run.tsbypasses it and importsturn-cursordirectly. Reduce the public API to lifecycle operations with small projections; keep SQL metadata and storage records private.- The claimed terminology/containment cut is incomplete. Public
TurnRecord/TurnSummarystill exposesessionId;continue-run.tsusessessionRecord; the supposedly thin cursor remains a 1,098-line renamed session implementation with a file-length exception; andstore.tsis a 536-line mirror with 27 forwards intostate.ts. This is containment by folder, not the understandable durable queue requested. NormalizeturnIdat the storage boundary and remove the forwarding/duplicate surfaces rather than documenting them as the final design.
What is solid: SQL schema/history are unchanged, durable prefix identity correctly ignores Pi's volatile assistant-envelope mutations while rejecting true branches, terminal writes retry, and CI is green. The PR is net -252 production lines, but the architecture and integration proof are not yet at the bar described in the PR and thread.
|
Quality pass at
Verified: 26 targeted tests, typecheck, lint/dependency rules, formatting, file-length, test-architecture, and diff checks. |
Co-Authored-By: David Cramer <david@sentry.io>
Co-Authored-By: David Cramer <david@sentry.io>
Co-Authored-By: David Cramer <david@sentry.io>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 7a0b5ec. Configure here.
Co-Authored-By: David Cramer <david@sentry.io>
Co-Authored-By: David Cramer <david@sentry.io>
Co-Authored-By: David Cramer <david@sentry.io>
Co-Authored-By: David Cramer <david@sentry.io>
Supersedes #1328 after #1344 moved `turn-session*` → turn-cursor / checkpoint. Stop threading `modelId` / skills / `reasoningLevel` through turn-cursor writes. Those fields were never restored from storage, so the write path was pretending they were checkpoint metadata. **Ownership (unchanged)** - checkpoint = resume boundary only (status, history boundary, slice) - profile + reasoning = turn route / history replacement - concrete model id = resolve at execution from profile config - skills = derive from history + catalog - channel + duration/usage = SQL conversation row **What changed** - Drop the three dead fields from `TurnRecord`, `upsertTurnRecord`, and checkpoint write args - Rename `runtimeMetadata` → `runtimeMetrics` for SQL-backed channel/metrics only - Remove the dead resume `existingSessionRecord.reasoningLevel` fallback - Explicit post-handoff contract: profile config, else inherited old route - No recovery-index policy changes Checked: `tsc --noEmit` (@sentry/junior + @sentry/junior-evals); focused vitest checkpoint + agent-resume + paused-turn **45/45**. Refs #1267 <!-- junior-request-attribution:start --> Requested by **U039RR91S**. <!-- junior-request-attribution:end --> <!-- junior-session-footer:start --> <!-- junior-conversation-id:slack%3AG6MCDB51U%3A1786211808.864869 --> -- [View Junior Session](https://junior-prod.sentry.dev/conversations/slack%3AG6MCDB51U%3A1786211808.864869) [[Sentry]](https://sentry.sentry.io/explore/conversations/slack%3AG6MCDB51U%3A1786211808.864869/?project=4510944073809921) <!-- junior-session-footer:end --> Co-authored-by: sentry-junior[bot] <264270552+sentry-junior[bot]@users.noreply.github.com> Co-authored-by: David Cramer <david@sentry.io>

Use one mailbox, one conversation lease, and one worker for durable conversation work. SQL conversation events keep the history. Redis stores temporary execution state in new v2 keys.
Reliability rules
A turn can continue after a timeout, retry, or yield only when its committed boundary changes. If the turn parks at the same boundary again, the worker stops the turn and records the error.
A process can stop while a turn runs. The next worker stops that turn. The user can start new work. Committed SQL history remains.
Queue work uses the conversation lease. OAuth can run outside the queue and uses the thread lock. One cursor lock controls each cursor write. A stale writer cannot change SQL or Redis after it loses the lock.
New Redis keys
This release reads and writes only
junior:conversation:v2:*andjunior:turn_cursor:v2:*. It does not read, move, or write old Redis state. Old mailbox, lease, and turn-cursor state can be lost. SQL schema and committed history do not change. Rollback support is out of scope.Integration tests
Production and the durable-queue tests use the same
createConversationWorkcomposition. The tests replace agent behavior and Slack HTTP. They use the realStateAdapterwith memory storage. They use an in-memory queue that implements the one-method queue interface.The tests cover these product rules:
Refs JUNIOR-62