Repository navigation
fix(runtime-host): isolate staged import recovery - #5407
Conversation
Generated-by: Codex
Astro-Han
left a comment
There was a problem hiding this comment.
Thanks for the fix. I read the diff against the surrounding code and agree the isolation is in the right place. One substantive gap (observability), plus a couple of nits.
What I verified
- Blast radius was real.
externalSessions.recover()is thestaterecovery phase of thesessiondomain module (packages/runtime-host/src/server/execution-composition.ts:2590), andrecoverRuntimeHostDomainModulesawaits every module per phase in order (packages/runtime-host/src/server/host-composition.ts:128-135), withstatefirst. So a rejectingrecover()aborted all later recovery phases and the Host never reachedready(host-kernel.ts:405). Catching at the coordinator loop is the minimal, correct authority for this — no protocol field or client state added. - Retry safety of leaving the Session at version 0.
RuntimeLedgerRepair.materializeTranscriptLedgerdocuments deterministic event ids with store-side dedupe precisely so "an interrupted import is resumable" (packages/runtime/src/runtime-ledger-repair.ts:62-71), andSessionManager.ensureTranscriptLedgeronly materializes whiletranscriptLedgerVersion !== 1(packages/runtime/src/session-manager.ts:4626-4647); itspreparedTranscriptLedgersmemo is per-process, so a restart re-attempts rather than skipping. Deferring the Session is safe to retry. - Not calling
requestDrain()is right.drainRequests() === 0in the new test matches intent:requestDrainpulls the composition out of startup, so requesting it from the first recovery phase would trade one startup failure for another. - The regression test is a real one. Without the
try/catch,recover()rejects atimported-1andimported-2is never prepared; the assertions (prepareAttempts == ['imported-1','imported-2'],discardAttempts == ['imported-1'],imported-1still0,imported-2at1) pin exactly the isolation contract in #5401. ThepublishStagingSessionextraction in the fixture is a faithful move of the previous default. - Docs. EN §10 and the zh-CN bullet describe the same contract as the code; nothing there claims behavior the code doesn't have.
Isolation is currently unobservable (see #5401: "The failure must remain observable")
packages/runtime-host/src/server/external-session-coordinator.ts:139-144
try {
await this.#prepareStagedSession(header.id);
} catch {
// One staged Session can remain unpublished for a later recovery
// attempt without preventing unrelated Sessions or Host startup.
}The error — and the session id, which is in scope as header.id — are dropped entirely. Consequences:
- A permanently broken staged import is now undiagnosable: every Host start pays one failed prepare plus one failed discard, and nothing is written anywhere. Before this change the double failure at least surfaced as
[runtime-host] startup failed:inpackages/runtime-host/src/candidate-entry.ts:88. - There is no counter, no log, and no client-visible signal, so issue reports for "this import never appears" have nothing to go on.
The repo already has the convention for exactly this shape — swallow to keep recovery going, but log: session-revision-coordinator.ts:195-203 logs [runtime-host] conversation copy cleanup deferred during recovery (${header.id}): …, and execution-composition.ts:2546 logs [runtime-host] ${name} will retry: …. Something like binding the error and logging [runtime-host] staged import recovery deferred (${header.id}): … would satisfy the issue's observability criterion without reintroducing the startup abort.
Nits
- nit: the
falsereturn from#prepareStagedSession(:368-378, prepare failed but discard succeeded) is also silently ignored by the loop. That was true before this PR, but the loop is now unconditional, so "staging deleted" and "staging retained" are indistinguishable in the recovery log. Two distinct messages would be cheap. - nit / question: with the Host now becoming
ready, a retained version-0 Session stays listed and is only gated at execution time byIMPORT_STAGING_UNAVAILABLE_REASON("Imported Session history is still being prepared." —packages/runtime-host/src/server/host-session-availability.ts:33,76,126), and the only re-attempt path is another Host restart. If that phantom entry is meant to be permanent, the design doc sentence is fine as written; if not, it may deserve a line about how it clears. - nit: the new test's
let fixture!self-reference works (the closure runs after assignment), but a small flag or apublishStagingSessionthat fails on demand would avoid the forward reference. Not worth churn on its own.
Astro-Han
left a comment
There was a problem hiding this comment.
The fix is correct: the catch sits at the right granularity inside the loop, a double-faulted staged Session keeps transcriptLedgerVersion: 0 and is retried by the next recover(), later staged Sessions still publish, and the test genuinely pins the invariant (reverting the try/catch fails it). Verified that residual version-0 Sessions do not wedge the later recovery phases — ledger recovery filters transcript-* invocations and execution recovery is a no-op on an empty ledger.
One P2 inline, one P3 in body.
P3 — test gap (recovery path, category 2): the new test pins "stays unpublished" but not the other half of the contract, that a later recover() can still finish the job. Flip the discard override to succeed after the first recover() and call it again, asserting the header reaches version 1.
| await this.#prepareStagedSession(header.id); | ||
| try { | ||
| await this.#prepareStagedSession(header.id); | ||
| } catch { |
There was a problem hiding this comment.
P2 (recovery path): this catch swallows the double failure with no signal at all. Version-0 staged Sessions are already invisible to the catalog, so a persistently failing import now retries silently on every restart and an operator has no way to discover a wedged staged Session. The codebase's own convention for this exact shape is a console.error line (session-revision-coordinator.ts logs conversation copy cleanup deferred during recovery). Please log sessionId and the error here.
There was a problem hiding this comment.
Fixed in b47214d. The recovery loop still isolates the failing staged Session and lets Host startup continue, but it now logs the Session id and a redacted failure message at the coordinator boundary. The regression asserts the diagnostic, the retained version-0 state, later Session recovery, and no drain request.
Keep staged Session recovery isolated when preparation and discard both fail, while logging the retained Session and a redacted diagnostic. Add regressions for observable failure and a later successful recovery attempt, and clarify the restart boundary in the design docs.\n\nGenerated-by: Codex
|
@Astro-Han Addressed in b47214d. Added a focused regression that fails preparation once, keeps the staged Session at version 0 after the first recover(), then succeeds on the next recover() and publishes version 1. The coordinator test suite passes 26/26; build, typecheck, format check, changed-file lint, and git diff --check also pass. The design docs now clarify that retry happens on a later Host startup recovery. |
Astro-Han
left a comment
There was a problem hiding this comment.
Both findings addressed, verified on the new head. The deferred-recovery log matches the codebase convention and goes one better by running the message through redactSecrets — worth keeping as the pattern for recovery-path logging. The new test pins the previously unpinned half: a retained staged Session retries and publishes on a later recover(). Nothing else outstanding.
Summary
Isolate startup recovery failures per staged external Session. If both Ledger preparation and staged Session cleanup fail, the Session remains unpublished for a later recovery attempt while the coordinator continues recovering later Sessions and allows Host startup to complete.
The Runtime Host coordinator remains the sole recovery authority; this adds no protocol fields or client recovery state. The external-session design documents now state the recovery contract.
Fixes #5401
Verification
npm run buildnode --test packages/runtime-host/dist/__tests__/external-session-coordinator.test.js— 25 passednpx biome lint packages/runtime-host/src/server/external-session-coordinator.ts packages/runtime-host/src/__tests__/external-session-coordinator.test.tsnpx biome format packages/runtime-host/src/server/external-session-coordinator.ts packages/runtime-host/src/__tests__/external-session-coordinator.test.ts docs/architecture/external-session-import-design.md docs/architecture/external-session-import-design.zh-CN.mdgit diff --checkThe full
@maka/runtime-hostsuite reached 1,943 passes and 0 failures, but exited 1 because six existingresumable-peer-stream.test.jstests were cancelled afterone-way blackhole triggers automatic recoveryleft a Promise pending. The same six cancellations reproduce when that unrelated test file runs alone.AI use
Tool(s) and scope: Codex(GPT 5.6 sol) implemented the coordinator recovery isolation, regression test, and design documentation update.
Checklist
Does this PR entail a change in behavior?