[Bug]: SSH reconnect kills a busy remote server, cancelling all running agents (2s reuse probe) #16477

Activity

1. juliusmarminge commented on Oct 6, 2026

   

   juliusmarminge

   on Oct 6, 2026

   Member

   More actions

   Note

   Grok responding on behalf of Julius.

   Triage

   Thanks @bloom-street for the overnight log analysis, the wake pattern and the clean SIGTERMs made this much easier to trace. Reading current main, a desktop SSH reconnect can stop a managed remote server, and the new process's startup recovery is what ends the running agents. It doesn't look like a crash or a provider bug.

   What I found

   • Stale local forward after sleep. ensureTunnelEntry probes the existing local forward for 2 seconds. After sleep that forward is dead, so the probe fails even if the remote server would have answered immediately. The tunnel finalizer then calls stopRemoteServer(), and the stop script SIGTERMs the recorded pid unless the host is marked external. The next launch starts a replacement. Host load isn't required for this path.
   • The elif ! wait_ready branch is a second kill. In packages/ssh/src/tunnel.ts, a managed pid that's still alive is SIGTERM'd when RUNNER_CHANGED=1, or when wait_ready misses REMOTE_REUSE_READY_TIMEOUT_MS (2s, each GET / capped at 1s). That runs when the finalizer didn't already remove the pid file. A loaded host can miss that budget. The logs don't show ssh.environment.tunnel.existing.stale vs. the launch script, so it's not clear which of the two fired each time; both produce the same SIGTERM and replacement.
   • Recovery on the new server. V2 orchestration recovery completed is startup reconciliation: stoppedSessions: 8 is every provider session not already stopped or errored, and run.background-work-cancelled clears background shells and subagents. continueThreadsAfterServerUpdate (off by default) would start a new continuation turn, but wouldn't preserve the killed provider process.
   • server-runtime.json case. The managed process is started with --base-dir ~/.t3, so the file it reads on the next launch is usually its own record. If that probe succeeds and state says managed, the script kills that pid and flips the host to external. fix(ssh): stop killing the managed server on reconnect #13341 and fix(ssh): reconnecting no longer restarts the remote server #13521 skip the kill only when the runtime pid is the managed pid; they don't change the timeout kill or the tunnel finalizer.
   • RUNNER_CHANGED case. This is the update path. Whether to defer an update while runs are active is a separate question for maintainers.
   • Related PRs and issues. fix(desktop): give the SSH readiness probe room for high-RTT links #7815 lengthens the desktop tunnel probe for high-RTT links and leaves the remote 2s reuse budget alone. fix(ssh): keep remote work running when the desktop closes #10951 would have stopped the finalizer from killing the remote on teardown but was closed unmerged. [Bug]: Restarting the desktop client breaks running remote Codex threads over SSH #10934 (quit/restart, same finalizer), SSH reconnect can replace an external T3 service with a competing managed runtime #5749 (a competing server beside an external service), and [Bug]: SSH remote environment never connects when link RTT exceeds ~1s, eg on in-flight wifi (readiness probe deadline is 1000ms) #7733 (desktop never connects because the local probe is 1s) are different failures.
   • Docs. The user docs say removing the connection stops a server T3 Code launched; a lid close isn't removing the connection, but the finalizer treats any stale tunnel as a managed-server stop.

   Likely fix area

   packages/ssh/src/tunnel.ts and the tunnel finalizer. Some options:

   • On a stale tunnel, wake, or client teardown, close only the local forward and keep stopRemoteServer() for explicit Disconnect / Remove.
   • In the launch script, avoid SIGTERMing a live managed pid just because a 2s loopback probe missed, e.g. retry with a budget closer to the 60s cold-start wait, or treat a live pid plus a bound port as enough to reuse.
   • Adopt the same-pid guard from fix(ssh): stop killing the managed server on reconnect #13341 / fix(ssh): reconnecting no longer restarts the remote server #13521 so a healthy server isn't killed when server-runtime.json names itself.

   Workaround: for a host that needs to outlive the laptop, npx t3 service install on the remote and then a normal paired connection avoids the managed-server lifecycle.

   A maintainer will decide on the fix direction.
2. juliusmarminge

   added

   bugSomething is broken or behaving incorrectly.Something is broken or behaving incorrectly.

   via-triageFiled through npx t3 triageFiled through npx t3 triage

   on Oct 6, 2026
3. topitopongsala commented on Oct 9, 2026

   

   topitopongsala

   on Oct 9, 2026

   More actions

   Same bug here, with both sides of the trace. This answers the open question in the triage: the kill came from the tunnel finalizer (stopRemoteServer), not from the launch script's wait_ready branch.

   Setup. Client: macOS desktop 0.0.46-nightly.20261007.2787. Remote: macOS, SSH environment, managed server in ~/.t3/ssh-launch/<key>/ (managed). The remote host was not overloaded. The client was on an outside network and not asleep.

   Client desktop.trace.ndjson (local time):

   08:55:17.384 EnvironmentSupervisor.connectionLost  target.kind=SshConnectionTarget connection.connected_ms=90152 connection.failure.reason=transport
   08:55:20.445 desktop.ipc.sshEnvironment.ensureEnvironment -> ssh/tunnel.ensureEnvironment -> ensureTunnelEntry
   08:55:20.459 shared.httpReadiness.waitForHttpReady  GET http://127.0.0.1:<local forward>/  x20, every ~100 ms, until 08:55:22.434
   08:55:22.461 ssh/tunnel.stopRemoteServer
   08:55:22.461 ssh/tunnel.closeTunnelEntry
   08:55:24.980 ssh/tunnel.launchOrReuseRemoteServer -> ensureTunnelEntry.create
   

   Remote side, same seconds:

   08:55:22.49  sshd: Accepted publickey from the client
   08:55:22.82  server.log: Relay client stopped
   08:55:28.0   every ACP agent (Devin) session/prompt fails with Interrupt; shutdown reconciliation marks the runs cancelled
   08:55:30.06  server.log: Listening on http://127.0.0.1:3774 (replacement)
   

   So a brief SSH transport drop is enough. The 2 s probe goes to the dead local forward, fails, and the finalizer stops a healthy remote server. That server was serving T3 Connect relay traffic the whole time. We saw 17 restarts like this in about 17 hours. Each one cancelled every running agent on the host.

   Two notes that may help pick the fix:

   • Ordering T3 Connect before SSH in the environment's routes does not remove the risk. The SSH route still runs ensureEnvironment when it is used as a fallback.
   • When more than one backend shares ~/.t3, each restart also cancels runs owned by the other backends ([Bug]: Stopping a duplicate backend cancels runs on the surviving server sharing the same T3 home #17003), so the damage reaches beyond the managed server.

   The first option in the triage fits what we saw: on a stale tunnel, close only the local forward, and keep stopRemoteServer() for an explicit Disconnect or Remove.
4. stockgeist commented on Oct 9, 2026

   

   stockgeist

   on Oct 9, 2026

   More actions

   Another way the launch script kills a healthy server on reconnect. The host here is not loaded and wait_ready succeeds; the kill comes from the managed branch of the server-runtime.json reuse block.

   What happened

   • Linux remote (desktop app also runs on it, serving 127.0.0.1:3773); a Mac connects over SSH.
   • Once, the Mac connected while the desktop app was closed, so the launch script started its own server (t3 serve --port 3774, state dir managed).
   • That server wrote itself into ~/.t3/userdata/server-runtime.json.
   • From then on, every reconnect killed it and started a new one: 99 Listening on http://127.0.0.1:3774 lines in ~/.t3/ssh-launch/<key>/server.log, each right after a tailscaled sh -l -s -- <key> session start. No OOM kills or core dumps. Every kill cancelled running agent turns and subagents.

   Why

   In REMOTE_LAUNCH_SCRIPT (still the same in 0.0.46-nightly.20261009.2861):

   if [ -n "$DEFAULT_REMOTE_PORT" ]; then
     REMOTE_PORT="$DEFAULT_REMOTE_PORT"
     if wait_ready "@@T3_REUSE_READY_TIMEOUT_MS@@"; then
       if [ "$REMOTE_MANAGED" = "managed" ]; then
         PID_TO_STOP="${REMOTE_PID:-$DEFAULT_RUNTIME_PID}"
         ...
           kill "$PID_TO_STOP" 2>/dev/null || true

   This branch is meant to retire the script's own server when some other default server is running. But when server-runtime.json points at the managed server itself, PID_TO_STOP == DEFAULT_RUNTIME_PID: the script kills the server it just found healthy, the following external wait_ready on that port fails, and it starts a new managed server, which writes itself into server-runtime.json again. So it loops on every reconnect.

   Suggested fix: when PID_TO_STOP equals DEFAULT_RUNTIME_PID (or the state dir's port equals DEFAULT_REMOTE_PORT), reuse the server and keep the state as managed.

   Workaround: write external to ~/.t3/ssh-launch/<key>/managed so the script reuses the server.

   Related: restarts on version change

   The RUNNER_CHANGED branch also kills a live managed server whenever the generated run-t3.sh differs, i.e. on every nightly update, regardless of running turns. Could these restarts wait (or be deferred) until no agent turns or subagents are running?

   Possibly related (not confirmed)

   While cleaning up, I SIGTERMed the stray 3774 server after the desktop app had already restarted on 3773. Afterwards server-runtime.json was gone (the userdata dir's mtime matches the SIGTERM), so the next SSH launch would have found no server and started a managed one again. It looks like a server removes server-runtime.json on shutdown without checking that the file still names its own PID. Restarting the desktop app rewrote the file.
5. luke2x

   added a commit that references this issue on Oct 11, 2026

   fix(ssh): keep restarting the managed server when the runner changes

   ...

   785eda8