fix(cli): record lost live messages on the run's terminal row - #3396
Open
ohdearquant wants to merge 7 commits into
Open
fix(cli): record lost live messages on the run's terminal row#3396ohdearquant wants to merge 7 commits into
ohdearquant wants to merge 7 commits into
Conversation
Under database-lock contention the live-message retry queue defers, then abandons its pending events at teardown. The count reached one line in one console tail while the session closed as a clean success, so the loss was invisible to every reader of the run. The session now carries a run.completed.message_loss reason, a summary naming how many events were lost, and evidence refs naming each queue that lost them, with the structured counts in the transition record. The completion-trust gate no longer demotes such a run to completed_empty either: that conclusion is read off the transcript, and this run's transcript is known to be missing part of itself. The loss survives both ways a terminal row is written by someone other than the leg that saw it. A leg that hands its terminal write to a resuming leg leaves the loss on the session, and the write that eventually happens adds up every deferred leg's count rather than reporting only its own, because each leg that observed a loss is gone by then and its queue with it. That accumulation holds across repeated deferrals: the deferred path is the only writer of a field it also reads, so it re-merges what is already there before writing, and a session that times out twice before anything terminal reports both. Recording the carried loss cannot cost the handoff, since the caller reads the deferring path's status to decide whether to resume: a bookkeeping write that fails is logged and the status stands rather than leaving a timed-out run unresumed. And a run that failed on its own keeps its own failure, with the loss appended to the evidence either way and claiming the reason code only when nothing else has.
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
added 5 commits
August 19, 2026 16:52
…ecided The invocation reducers looked for message-loss evidence inside their all-completed arm. Every arm above it returns first, so a child that lost part of its transcript and a sibling that failed produced an invocation row with no trace of the loss -- the case that most needs both. The scan moves ahead of the precedence ladder in both reducers; only the reason code still defers, because a row carries one. The scheduler's reducer is the one scheduled runs go through and it had no message-loss arm at all, so an all-completed set mapped to a clean pass however incomplete a child's transcript was. It now returns the same reason the flow reducer does. The evidence predicate moves next to the reason code it justifies so both reducers read the same one instead of one owning it privately. The carried-loss validator also drops queue entries whose count is zero or negative, rather than summing them into totals like "-3 event(s) lost".
…rminal write The loss was recorded only inside the guarded terminal transition. Every path that does not write -- a concurrent teardown winning the compare-and-swap, the row already terminal on arrival, a transition rejected under the guard -- returned the winner's status and dropped the loss with it, so the terminal row read as a clean run while part of the transcript was gone. Those paths now leave the loss on the session instead. The winner's status and reason are untouched, since an earlier terminal record stands; only the loss is kept, in the same place a deferred leg leaves it. The evidence predicate the invocation reducers use reads that carrier too, so the loss reaches a reader rather than sitting in a column nothing consults.
…owns keep both Two teardowns of one session can each hold live-message loss and both lose the terminal write. Each read the row and rewrote one shared node_metadata value, so whichever wrote second replaced the other leg's record and the session undercounted what its transcript is missing. Every observing leg now writes a key of its own: the database merges a patch key by key, so the records add up instead of colliding, and neither path is left doing a read-modify-write. A leg that does not own the terminal row also records only what it saw. The earlier legs' losses are already on the row under their own keys, so the merged view it would otherwise write counted them a second time.
Keep-both on the CHANGELOG Fixed section: this branch bullet and the subprocess-abandon bullet from main.
…l child Two readers decided whether a session lost live messages. One asked only whether the row carried a non-empty queue list; the other validated each entry before summing it. A carried payload with `lost: 0`, a negative count, a boolean, or no owner therefore made a run report message loss with nothing to count. Both now go through one predicate, so a payload one rejects cannot produce a loss verdict from the other. The flow reducer's precedence ladder claims no outcome when a child exists but has not reached a terminal status, and the fallback that then decides it was the clean-pass reason code. A run whose transcript is known to be incomplete was reported as a run that went fine. That fallback now reports the loss, the same way every arm above it does.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Under database-lock contention the live-message retry queue defers, then abandons its pending events at teardown. The count reached one line in one console tail while the session closed as a clean success, so the loss was invisible to every reader of the run.
The session now carries a
run.completed.message_lossreason, a summary naming how many events were lost, and evidence refs naming each queue that lost them, with the structured counts in the transition record. The completion-trust gate no longer demotes such a run tocompleted_emptyeither: that conclusion is read off the transcript, and this run's transcript is known to be missing part of itself.The loss survives a terminal row written by someone else
Two ways that happens, and both are covered.
A leg that hands its terminal write to a resuming leg leaves the loss on the session, and the write that eventually happens adds up every deferred leg's count rather than reporting only its own. The leg that observed each loss is gone by then, and its queue with it.
That accumulation holds across repeated deferrals. The deferred path is the only writer of a field it also reads, so it re-merges what is already stored before writing. A session that times out twice before anything terminal reports both losses; previously the second deferral overwrote the first and the terminal row silently reported a smaller number than the truth. Reproduced at three legs losing four events each: eight reported out of twelve.
Recording the carried loss cannot cost the handoff. The caller reads the deferring path's status to decide whether to resume, so a bookkeeping write that fails is logged and the status stands, rather than trading a run that resumes for one that hangs unresumed.
A run that failed on its own keeps its own failure. The loss is appended to the evidence either way and claims the reason code only when nothing else has.
Tests
Deferred-then-terminal, deferred-twice-then-terminal, a deferred leg that lost nothing (without which a resumed write could report a loss on every deferred session), a loss on a run that failed for its own reasons, and a failed record not costing the handoff. Plus pins on what a drifted carried payload is allowed to do to the count: a non-list
queuesfield is dropped rather than walked, and a queue entry with a non-numericlostis dropped rather than summed.Removing the re-merge from the deferred path reddens the multi-leg test and nothing else in the file.
Relationship to the earlier attempt
This supersedes an earlier PR carrying the same change, re-cut with a single parent on current
mainso the repeated changelog collisions with the landing cluster dissolve. The tree is that branch's head plus the multi-leg accumulation fix and its regression test.Verification bound
The concurrent-teardown test runs against SQLite, so what it exercises is the
json_patchmerge. The Postgresjsonb ||path for the same merge is covered by the dialect-parity tests intests/state/test_dual_backend.py, which start a container and skip when Docker is unavailable. They skipped here, oncould not start Postgres container, so that path is not one I have observed pass on this machine.Design notes on the per-leg loss records
Each teardown that observed loss writes its own key rather than rewriting a shared one, which is what removes the lost update. Three consequences worth naming, since each is a deliberate trade rather than an oversight.
Cumulative teardown work is now linear rather than quadratic: a leg writes only what it saw, with no read of the accumulated list, and the whole set is read once at the terminal write.
The keys are not compacted after that terminal write folds them in. Clearing them would race the next teardown's write, which is the lost update this change exists to remove. The growth is bounded by the number of teardowns that actually lost messages rather than by teardowns: both write sites are guarded on a non-empty loss, so a session that never loses a message never writes a key.
The assistant-output lookup still runs when a loss is present, even though the loss already decides the branch below it. Its result is persisted as
has_assistant_output, so gating it would leave that field absent on exactly the runs whose transcript is known to be incomplete, which are the runs where a reader most needs it.