Repository navigation
fix(tui): a dead terminal must not take the proxy with it - #410
Conversation
KarpelesLab#404 flipped stdout non-blocking so a paint could not block the event loop. That fixed the stalls, but moved stdout's write failures onto the asynchronous path, where they arrive as an 'error' event on the stream. Nothing listens for it, so Node promotes it to an uncaughtException and installCrashHandlers exits: the proxy now dies whenever its terminal goes away — a pane closed, a pty recreated — and on the way out it orphans any sidecar it supervises, which then holds the port against the replacement process. Observed twice within a day on a live fork deployment, both `write EPIPE` at WriteWrap.onWriteComplete, the first thirteen minutes after that change first ran. The crash log had been armed for six weeks with no entries before it. A display may no more kill the proxy than block it: listen for the error, record that stdout is gone, and stop painting, while everything else keeps serving. The exit sequence becomes best-effort, because a terminal that has already gone will fail it, and the listener outlives that write so a late asynchronous failure is absorbed rather than thrown. index.js already used this idiom for the status command's EPIPE; the TUI never needed it until stdout stopped blocking. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The guard added in the previous commit had a hole. stop() removed the stdout
error listener immediately after writing the exit sequence, with a comment
claiming the listener outlived that write. It does not: the removal is
synchronous and the failure is not. Flipping back to blocking does not make
writes ALREADY QUEUED synchronous either, and shutdown runs well past that point
— stopping background workers, then awaiting a state save. A paint still in
flight fails into that window with nothing listening, and the proxy died of it
a third time. The listener now lives as long as the process; it only sets a
flag, and start() runs once.
The handler itself no longer treats a broken pipe as fatal, which is the fix
that covers every path rather than one teardown ordering. A terminal going away
says nothing about the proxy's state — accounts, routes and inflight requests
are exactly as they were — so ending the process punishes every routed session
for a closed pane. Recorded once and survived; a genuine uncaught exception
still reports and exits non-zero. reportFailure in server.js already described
this hazard ("this daemon treats an uncaught EPIPE as fatal"); this makes that
no longer true.
Entries now carry err.code and err.syscall: "Error: write EPIPE" alone names
neither the stream nor the call, which is what made this take three occurrences
to diagnose.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Pushed a follow-up commit: the patch as originally submitted had a hole, and I'd rather you saw the corrected version than the one that bit me.
That killed the proxy a third time, on the deployment I took the original evidence from: Two changes:
If you would rather keep Gate on this branch: |
…y on SIGHUP The stdout 'error' guard now says what happened, once, through the console.error saved before the TUI patched it, and render() returns before composing a frame for a terminal nobody can see. stdin gets the same guard; an attach client quits when its terminal dies, while the server keeps serving headless. stop() releases the stdout guard only when its final write completes cleanly, so a late EPIPE during shutdown is still absorbed, and SIGHUP now runs the same shutdown funnel as SIGINT/SIGTERM so a closed pane persists quota state. The blanket EPIPE absorption in crash-log.js is dropped: it silently swallowed a broken pipe from any stream, against the daemon's rule that an uncaught error is fatal, and the TUI's own listeners make it redundant. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Thirty-six commits since 1.1.20. Several change what a client sees on a failure, so read the first section before upgrading a shared deployment. Behaviour changes #439 a 401 from upstream is no longer relayed to the client: the request fails over like a 403, and an API-key account or an OAuth account with no refresh token leaves rotation (it used to be picked again on every request). With nothing left the client gets the proxy's own error #439 a path under /teamclaude/ that no control route claims — a typo, or the wrong verb — answers 404 locally instead of being forwarded upstream under a fleet credential #429 the synthetic 429's retry-after is the real reset of the windows blocking the request's candidate accounts, not a flat 60s, and the message counts only those candidates; #408 names accounts that need a re-login instead of calling them "at quota" #438 session pins are per conversation (session id plus a digest of the first message), so a session's subagents spread across accounts. `sessions.items[].id` in status is the composite key, load is counted per conversation, and a persisted concurrency cap re-learns #378 a `thread: continue` bound for a per-account third-party upstream is refused with the 400 Anthropic gives, so the client resends the whole conversation; `messageThreads: true` opts a relay out #434 a 429 whose x-codex-* headers show a spent account-wide window holds the account like an Anthropic rejection; a spent model-scoped bucket only moves the request #437 a Codex response head is awaited for five minutes (Anthropic unchanged) #411 idle keep-alive connections are held 120s on both listeners #389 with session distribution on, requests carrying no session id rotate on a cursor of their own instead of all resting on the current account #405 `defaultClientMode` ("mitm" | "base-url") sets what `run` and `env` do without a flag; `--mitm` / `--no-mitm` decide per launch, and in base-URL mode `env` unsets a stale proxy export naming this proxy #439 route `--bucket` is validated; an array `switchThreshold` reads as the default with one line saying so Features #419 an MCP management endpoint at POST /teamclaude/mcp, off unless `proxy.mcp` is "read" or "full"; a named client key is read-only even in full mode, and with no proxy key it serves only this machine #428 per-account `switchThreshold`, a number or a per-bucket table #406 a Claude+Codex pool is drawn as two panes on a wide terminal, each with its own current marker; #392 names the provider in a mixed list; #418 lets the operator arrange the list (`displayOrder`); #376 draws loopback-served accounts last; #435 shows the percentage beside a bar's countdown; #394 shows the running version in the header #430 free Codex rate-limit reset credits in status, the TUI and the dashboard #385 `proxy.terminalOnly` tunnels chatgpt.com so ChatGPT Desktop stays out Fixes #404 a TUI paint can no longer block the proxy (stdout non-blocking, frames dropped while the terminal is behind); #410 a dead terminal no longer takes the proxy with it, and SIGHUP shuts down cleanly #433 token usage is booked from Codex Responses streams #386 #387 #388 the Codex five-hour window is read from the model-scoped family and the usage probe, and extra limits are named from their entries #431 a headerless 429 that follows the request is retried once #432 #439 startup and collaborator log lines reach the TUI's activity pane and log file instead of the covered terminal #415 #403 #439 reload mirrors `priority`, `disabled`, `stripRequestFields` onto the config entry, and a reload during a removal does not re-add it #439 the Host check uses the address actually bound; sx.org calls time out #381 the dashboard polls status before asking for a key #403 session outcome accounting classifies the decoded path; account names in daemon log lines are sanitised Tooling #371 #372 #373 `npm run typecheck` (tsc over the JS sources) in CI, with a strict-mode ratchet: per-file strict diagnostics may not grow past the pre-merge commit (2006 at introduction, 1735 now) #401 docker workflow actions bumped Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Takes KarpelesLab/teamclaude up to ba01b4f while keeping the fork's routing safeguards (403 cooldown without parking, send-failure fail-over, outbound content-length, the single predispatch wait budget, cappedMessage, the overload slot release, the dead-refresh-token guard and refresh retry, the session home wait from d7810f1, and unranked-priority semantics). Adopted from upstream, among others: per-conversation session pins (KarpelesLab#438), 401 fail-over without parking (KarpelesLab#439, KarpelesLab#473), real quota-reset retry-after and candidate counts (KarpelesLab#429, KarpelesLab#408), a headerless 429 retry (KarpelesLab#431), fail-over on a failing 200 stream (KarpelesLab#470), per-account egress proxies (KarpelesLab#441), extra-usage fallback (KarpelesLab#427), maxSpend (KarpelesLab#466), stripOverageHeaders (KarpelesLab#478), keep-alive that outlives the client pool (KarpelesLab#411), a dead terminal not killing the proxy (KarpelesLab#410), console resolution per call (KarpelesLab#432), config reload and sync fixes (KarpelesLab#465, KarpelesLab#415), and the dashboard, TUI and Codex work since 1.1.20. Integration fixes the merge needed beyond conflict hunks: - upstream request-path code that referenced upstream-only locals (sx, route, ctx.tried) rewritten for the fork's forwardRequest - resolveSwitchThreshold was declared twice after a clean auto-merge - the fork's warm-up probe now forwards a routed account through its own proxy instead of sending its credential direct (new regression test) - a routing failure during a token refresh arms the routing hold instead of parking the account; a pinned request may use an account on routing hold - the headerless-429 branch no longer writes after headers were sent - canonical-state allowlists widened for upstream's new quota fields; saves use exportState() - the fork's home wait keys on the conversation pin like selection does Tests adapted where the fork deliberately differs (warm-up probe on by default, account-anchored TUI cursor, coordinator-built Prober and Warmer, unranked priority, per-conversation pin keys, fail-over-only 401, the wait budget instead of inline waits), each with an in-file note. Full suite 2919/2919, lint, typecheck and the strict ratchet (1411 vs 1465) pass. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
"Error: write EPIPE" on its own names neither the stream nor the call, which is what made a broken pipe take three occurrences to pin down. Entries now carry err.code and err.syscall beside the kind. The first version of this also absorbed a broken pipe process-wide. Upstream KarpelesLab#410 dropped that deliberately: it swallowed a broken pipe from any stream, against the daemon's rule that an uncaught error is fatal, and the TUI's own stdout/stdin listeners already cover the terminal case. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Follow-up to #404, which I authored: that change fixed the stalls but introduced a crash. This fixes it.
What #404 did, and what it cost
Flipping stdout non-blocking stopped a full-screen paint from holding the event loop. It also moved stdout's write failures onto the asynchronous path, where a failure arrives as an
'error'event on the stream rather than throwing at the call site. Nothing listened for it, so Node promotes it to anuncaughtException,installCrashHandlersrecords it and callsexit(1).The result: the proxy dies whenever its terminal goes away — a pane closed, a pty recreated, a terminal emulator restarted. Worse, the hard exit skips
stop(), so a supervised sidecar is orphaned and keeps its port, and the replacement server then cannot bind (Address already in use) and restart-loops indefinitely.Evidence
Two crashes within a day on a live deployment of my fork, both identical:
WriteWrap.onWriteCompleteis the async write-completion path, which only exists once the stream is non-blocking. The crash log (added in #150) had been armed for six weeks with no entries; its first is thirteen minutes after #404 first ran.The fix
A display may no more kill the proxy than block it.
start()installs an'error'listener before the first write and records that stdout is broken._paint()returns early when it is, so the process serves on without a display — and a stream that never drains cannot strand the pending-paint handshake.stop()makes the exit sequence best-effort, since a terminal that has already gone will fail it, and releases the listener after that write so a late async failure is still absorbed.index.jsalready carried this idiom for the status command's EPIPE; the TUI never needed it until stdout stopped blocking.Verification
npm test— 1868 pass, 0 fail (3 new regression tests).npm run typecheck— clean.npm run typecheck:strict -- --base upstream/master—1831 strict-mode diagnostics (base 7b1e92d0: 1831), no regression.npm run lint— clean.The new tests drive the real failure: emit
'error'on stdout and assert that painting stops and nothing throws, including when a frame is already parked awaitingdrain.🤖 Generated with Claude Code