[Bug]: t3 project CLI treats any 1s live-server probe failure as "no server": deletes server-runtime.json and writes offline behind a running server #7504

What happened

While bulk-registering projects on a Windows machine I read how t3 project add|remove|rename decides between talking to the running server and writing to state.sqlite directly, and found that the live-server probe is bounded to 1 second and that any probe failure (timeout, auth error, undeclared status) is treated the same as "no server": the CLI deletes the running server's server-runtime.json and then applies the mutation offline against the database the live server still has open.

Diagnosis

Grounded in the installed nightly bundle (apps/server/dist/bin.mjs, //#region src/cli/project.ts):

• PROJECT_CLI_LIVE_SERVER_TIMEOUT = seconds(1); fetchLiveOrchestrationSnapshot and dispatchLiveOrchestrationCommand are both wrapped in withProjectCliLiveServerTimeout.
• tryResolveLiveProjectExecutionMode reads server-runtime.json, calls fetchLiveOrchestrationSnapshot, and on any failure does:

  logDebug("Failed to connect to the persisted project CLI server.", ...)
  yield* clearPersistedServerRuntimeState(config.serverRuntimeStatePath)
  return none()
  

  It does not distinguish ECONNREFUSED (server really gone) from a timeout / 401 / 5xx (server alive but slow or rejecting).
• runProjectMutation then builds ProjectCliRuntimeLive (its own OrchestrationEngine over the same state.sqlite) and dispatches the command in mode: "offline".

Consequences when the probe fails against a live server:

1. The running server's server-runtime.json is deleted, so t3 pair, t3 connect and later t3 project calls can no longer discover it until the server is restarted (the server only writes the file at startup).
2. The project mutation is appended to the event store behind the live server's back. The live engine's in-memory read model is not updated, so the project does not appear in the open desktop/web UI until restart, even though the CLI printed "Added project ...".
3. Two OrchestrationEngine instances (live + offline) write the same event store and projection_state concurrently.

A 1 s bound is easy to miss on Windows: server.getConfig is already known to block for up to 5 s on this platform (#4697, #5137), and orchestration.snapshot grows with projects/threads (64 projects here).

What is verified vs. derived: the code path, constants and the unconditional clearPersistedServerRuntimeState are read directly from the shipped bundle and match src/cli/project.ts on main. I did not force the timeout empirically on this machine (my bulk registration was done with the desktop app fully closed, precisely to avoid this path); the consequences above follow from the code.

Suggested fix (small): only clear the runtime file and fall back to offline when the connection is refused or the recorded pid is not alive; on timeout/auth/undeclared-status errors fail loudly ("live server at did not answer within 1s; retry, or stop the server and rerun") instead of silently writing offline. Optionally raise the bound to the 5 s used by CLOUD_CLI_LIVE_SERVER_TIMEOUT.

Steps to reproduce

Deterministic repro requires making orchestration.snapshot exceed 1 s; by construction:

1. Run the desktop app (server writes <T3 home>/userdata/server-runtime.json).
2. Make the server slow to answer orchestration.snapshot for >1 s (large project/thread count, or pause the server process for 2 s under a debugger).
3. npx t3 project add <some-repo> (or node apps/server/dist/bin.mjs project add ...).
4. Observe: CLI prints "Added project ..." in offline mode, server-runtime.json is gone, and the project is absent from the running UI until restart.

Version

0.0.34-nightly.20260819.1132 (commit 36f4314); same code on main today.

Environment

Windows 11 Pro 10.0.26100, T3 Code (Nightly) desktop, Node 24.11.1.

Evidence

# installed bundle, src/cli/project.ts region
const PROJECT_CLI_LIVE_SERVER_TIMEOUT = seconds(1);
const withProjectCliLiveServerTimeout = (effect) => effect.pipe(timeout(PROJECT_CLI_LIVE_SERVER_TIMEOUT));
const fetchLiveOrchestrationSnapshot = (origin, bearerToken) => gen(function* () { ... }).pipe(withProjectCliLiveServerTimeout, mapError(projectCommandErrorFromLiveServerRequest));
const tryResolveLiveProjectExecutionMode = fn$1("tryResolveLiveProjectExecutionMode")(function* (environmentAuth, config) {
  const runtimeState = yield* readPersistedServerRuntimeState(config.serverRuntimeStatePath);
  if (isNone(runtimeState)) return none();
  const attempted = yield* result$1(withProjectCliSessionToken(environmentAuth, (token) => fetchLiveOrchestrationSnapshot(runtimeState.value.origin, token).pipe(...)));
  if (attempted._tag === "Success") return some(attempted.success);
  yield* logDebug("Failed to connect to the persisted project CLI server.", { origin: runtimeState.value.origin, cause: attempted.failure });
  yield* clearPersistedServerRuntimeState(config.serverRuntimeStatePath);   // <- unconditional
  return none();
});

Related observation (not the subject of this issue): after a clean quit of the Windows desktop app (CloseMainWindow on the main window; all T3 Code (Nightly).exe processes exited) server-runtime.json was left behind containing the dead server pid, i.e. the server's release finalizer that normally clears it (//#region src/cli/server.ts, persistServerRuntimeState / clearPersistedServerRuntimeState) did not run on this platform. A stale file is exactly what makes the offline fallback above attractive, but it is also what makes the "any failure" heuristic risky: the CLI cannot tell a stale file from a live-but-slow server.

Related issues

#5749 / #6097 (competing runtimes against the same database) are about the service/desktop split, not the project CLI; no existing issue mentions the 1 s probe or the runtime-file deletion.

Fix applied or workaround

None needed for me: I only used the offline path with the desktop fully closed. Workaround for others: quit T3 completely before using t3 project ..., or check that the project appears in the UI afterwards.

Filed by

Claude (Fable 5) via Claude Code on the reporter's machine, at the reporter's request.