[Bug]: Saved thread is reported missing and cannot be deleted, archived, or continued after two servers share its database #14353

Activity

1. juliusmarminge commented on Sep 30, 2026

   

   juliusmarminge

   on Sep 30, 2026

   Member

   More actions

   Triage

   Confirmed on current main. This is a stuck-thread bug, separate from open issue #6097.

   Two servers can share one T3 home because desktop startup only checks ports. resolveDesktopBackendPort in apps/desktop/src/app/DesktopApp.ts walks up from 3773 until it finds a free port, then starts a backend on the default home anyway. Server startup writes server-runtime.json without checking whether another live process already owns that file (apps/server/src/server.ts). That split is #6097. This issue is about what the command engine does after it happens.

   OrchestrationEngine loads its command read model once, after projection bootstrap, from ProjectionSnapshotQuery.getCommandReadModel() (apps/server/src/orchestration/Layers/OrchestrationEngine.ts). thread.delete, thread.archive, thread.settle, thread.turn.start, thread.turn.interrupt, and thread.session.stop all go through requireThread, which checks that in-memory list (apps/server/src/orchestration/commandInvariants.ts). The error in the report is that invariant failing.

   The UI doesn't use that list. Shell and thread snapshots read the projection tables on every request (apps/server/src/orchestration/http.ts), so the other server's commits stay visible while every mutation is rejected and stored as a command receipt.

   Restarting works because startup reads the projection tables again. Stopping the extra server isn't enough, because the surviving process keeps its stale model.

   The failure path doesn't reliably repair it. After a rejection, reconcileReadModelAfterDispatchFailure replays orchestration_events rows whose sequence is greater than the in-memory snapshotSequence, up to 1000 rows. It doesn't reload projections and doesn't look backward. sequence is SQLite AUTOINCREMENT, so if this process appends any event after the other server inserted thread.created, projectEvent moves snapshotSequence past that row and the creation event is skipped. From then on, delete, archive, settle, start, interrupt, and session-stop keep failing until the process restarts. A projector decode error during replay is swallowed, which also leaves the model unchanged. And retrying with the same command id returns "previously rejected" without replaying at all.

   That replay is also the wrong recovery even when it does catch up: it publishes the other process's events onto this process's event bus. #6097 already has a duplicate provider-session report that comes from this path. When a thread is missing, catch-up should refresh the command model from the projection tables (the same read a restart does) and shouldn't feed those events to provider reactors.

   #6097 should still stop a second server from opening the database. This issue should stay open for the recovery gap: a thread that exists in the projection, with a creation event and no deletion, shouldn't stay permanently missing in the process that's still serving it.
2. juliusmarminge

   added

   bugSomething is broken or behaving incorrectly.Something is broken or behaving incorrectly.

   via-triageFiled through npx t3 triageFiled through npx t3 triage

   on Sep 30, 2026
3. praveenperera

   closed this as completedon Sep 30, 2026
4. praveenperera

   reopened this on Oct 1, 2026
5. juliusmarminge commented on Oct 2, 2026

   

   juliusmarminge

   on Oct 2, 2026

   Member

   More actions

   Thanks for taking the time to report this and provide the details. We revisited it during the orchestrator V2 cleanup.

   The specific permanent missing-thread invariant depended on V1 startup-loaded command state and skipped event catch-up. V2 delete and mutation commands read ProjectionStore records when dispatched, removing that exact command-model split. Shared-home multi-process safety remains a separate concern; do not claim every two-server symptom fixed.

   I’m closing this report because the implementation it targets has been replaced. That does not mean every similar symptom is fixed.

   Source reviewed.

   Invite a fresh V2 reproduction. This retires the V1 stale in-memory command-state diagnosis only; no two-server runtime test performed.

   If you still hit this on a current build, please reply with the app/server versions and the steps that reproduce it. We can reopen this if the original problem is still there.
6. juliusmarminge

   closed this as not plannedon Oct 2, 2026