Skip to content

fix(tui): a dead terminal must not take the proxy with it - #410

Merged
MagicalTux merged 4 commits into
KarpelesLab:masterfrom
rikbrown:rik/tui-epipe-guard-upstream
Sep 19, 2026
Merged

MagicalTux merged 4 commits into
KarpelesLab:masterfrom
rikbrown:rik/tui-epipe-guard-upstream

Conversation

@rikbrown

Copy link
Copy Markdown
Contributor

Follow-up to #404, which I authored: that change fixed the stalls but introduced a crash. This fixes it.

What #404 did, and what it cost

Flipping stdout non-blocking stopped a full-screen paint from holding the event loop. It also moved stdout's write failures onto the asynchronous path, where a failure arrives as an 'error' event on the stream rather than throwing at the call site. Nothing listened for it, so Node promotes it to an uncaughtException, installCrashHandlers records it and calls exit(1).

The result: the proxy dies whenever its terminal goes away — a pane closed, a pty recreated, a terminal emulator restarted. Worse, the hard exit skips stop(), so a supervised sidecar is orphaned and keeps its port, and the replacement server then cannot bind (Address already in use) and restart-loops indefinitely.

Evidence

Two crashes within a day on a live deployment of my fork, both identical:

=== 2026-09-15T21:58:50.452Z uncaughtException ===
Error: write EPIPE
    at WriteWrap.onWriteComplete [as oncomplete] (node:internal/stream_base_commons:87:19)

WriteWrap.onWriteComplete is the async write-completion path, which only exists once the stream is non-blocking. The crash log (added in #150) had been armed for six weeks with no entries; its first is thirteen minutes after #404 first ran.

The fix

A display may no more kill the proxy than block it.

  • start() installs an 'error' listener before the first write and records that stdout is broken.
  • _paint() returns early when it is, so the process serves on without a display — and a stream that never drains cannot strand the pending-paint handshake.
  • stop() makes the exit sequence best-effort, since a terminal that has already gone will fail it, and releases the listener after that write so a late async failure is still absorbed.

index.js already carried this idiom for the status command's EPIPE; the TUI never needed it until stdout stopped blocking.

Verification

  • npm test — 1868 pass, 0 fail (3 new regression tests).
  • npm run typecheck — clean.
  • npm run typecheck:strict -- --base upstream/master — 1831 strict-mode diagnostics (base 7b1e92d0: 1831), no regression.
  • npm run lint — clean.

The new tests drive the real failure: emit 'error' on stdout and assert that painting stops and nothing throws, including when a frame is already parked awaiting drain.

🤖 Generated with Claude Code

rikbrown and others added 2 commits September 16, 2026 10:22
KarpelesLab#404 flipped stdout non-blocking so a paint could not block the event loop.
That fixed the stalls, but moved stdout's write failures onto the asynchronous
path, where they arrive as an 'error' event on the stream. Nothing listens for
it, so Node promotes it to an uncaughtException and installCrashHandlers exits:
the proxy now dies whenever its terminal goes away — a pane closed, a pty
recreated — and on the way out it orphans any sidecar it supervises, which then
holds the port against the replacement process.

Observed twice within a day on a live fork deployment, both `write EPIPE` at
WriteWrap.onWriteComplete, the first thirteen minutes after that change first
ran. The crash log had been armed for six weeks with no entries before it.

A display may no more kill the proxy than block it: listen for the error, record
that stdout is gone, and stop painting, while everything else keeps serving. The
exit sequence becomes best-effort, because a terminal that has already gone will
fail it, and the listener outlives that write so a late asynchronous failure is
absorbed rather than thrown. index.js already used this idiom for the status
command's EPIPE; the TUI never needed it until stdout stopped blocking.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The guard added in the previous commit had a hole. stop() removed the stdout
error listener immediately after writing the exit sequence, with a comment
claiming the listener outlived that write. It does not: the removal is
synchronous and the failure is not. Flipping back to blocking does not make
writes ALREADY QUEUED synchronous either, and shutdown runs well past that point
— stopping background workers, then awaiting a state save. A paint still in
flight fails into that window with nothing listening, and the proxy died of it
a third time. The listener now lives as long as the process; it only sets a
flag, and start() runs once.

The handler itself no longer treats a broken pipe as fatal, which is the fix
that covers every path rather than one teardown ordering. A terminal going away
says nothing about the proxy's state — accounts, routes and inflight requests
are exactly as they were — so ending the process punishes every routed session
for a closed pane. Recorded once and survived; a genuine uncaught exception
still reports and exits non-zero. reportFailure in server.js already described
this hazard ("this daemon treats an uncaught EPIPE as fatal"); this makes that
no longer true.

Entries now carry err.code and err.syscall: "Error: write EPIPE" alone names
neither the stream nor the call, which is what made this take three occurrences
to diagnose.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@rikbrown

Copy link
Copy Markdown
Contributor Author

Pushed a follow-up commit: the patch as originally submitted had a hole, and I'd rather you saw the corrected version than the one that bit me.

stop() removed the stdout 'error' listener immediately after writing the exit sequence, with a comment asserting that the listener outlived that write. It does not — the removal is synchronous and the failure is not. Flipping back to blocking also does not make writes already queued synchronous, and shutdown continues well past that point (background workers, then an awaited state save). A paint still in flight fails into that window with nothing listening.

That killed the proxy a third time, on the deployment I took the original evidence from:

=== 2026-09-16T15:19:12.435Z uncaughtException ===
Error: write EPIPE
    at WriteWrap.onWriteComplete [as oncomplete] (node:internal/stream_base_commons:87:19)

Two changes:

  • The listener now lives as long as the process. It only sets a flag, and start() is called once, so nothing accumulates.
  • crash-log.js no longer treats an uncaught EPIPE as fatal. This is the fix that generalises: a dead terminal says nothing about the proxy's state, so patching teardown orderings one at a time is chasing the symptom. It is recorded once — with err.code/err.syscall, since a bare "Error: write EPIPE" names neither the stream nor the call — and then survived. A genuine uncaught exception still reports and exits non-zero. reportFailure in server.js already documents this hazard ("this daemon treats an uncaught EPIPE as fatal"); this makes that no longer true.

If you would rather keep crash-log.js strictly faithful to Node's exit-on-uncaught behaviour, say so and I will confine the change to the TUI listener — but then a headless server writing to a closed stdout still dies, so I think the broader fix is the right one.

Gate on this branch: npm test 1872 pass / 0 fail, typecheck clean, typecheck:strict --base upstream/master 1831 vs base 1831, lint clean.

MagicalTux and others added 2 commits September 20, 2026 08:35
…y on SIGHUP

The stdout 'error' guard now says what happened, once, through the console.error
saved before the TUI patched it, and render() returns before composing a frame
for a terminal nobody can see. stdin gets the same guard; an attach client quits
when its terminal dies, while the server keeps serving headless. stop() releases
the stdout guard only when its final write completes cleanly, so a late EPIPE
during shutdown is still absorbed, and SIGHUP now runs the same shutdown funnel
as SIGINT/SIGTERM so a closed pane persists quota state.

The blanket EPIPE absorption in crash-log.js is dropped: it silently swallowed a
broken pipe from any stream, against the daemon's rule that an uncaught error is
fatal, and the TUI's own listeners make it redundant.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@MagicalTux
MagicalTux merged commit 784d75c into KarpelesLab:master Sep 19, 2026
5 checks passed
@MagicalTux MagicalTux mentioned this pull request Sep 20, 2026
MagicalTux added a commit that referenced this pull request Sep 20, 2026
Thirty-six commits since 1.1.20. Several change what a client sees on a
failure, so read the first section before upgrading a shared deployment.

Behaviour changes
  #439 a 401 from upstream is no longer relayed to the client: the request
       fails over like a 403, and an API-key account or an OAuth account with
       no refresh token leaves rotation (it used to be picked again on every
       request). With nothing left the client gets the proxy's own error
  #439 a path under /teamclaude/ that no control route claims — a typo, or the
       wrong verb — answers 404 locally instead of being forwarded upstream
       under a fleet credential
  #429 the synthetic 429's retry-after is the real reset of the windows
       blocking the request's candidate accounts, not a flat 60s, and the
       message counts only those candidates; #408 names accounts that need a
       re-login instead of calling them "at quota"
  #438 session pins are per conversation (session id plus a digest of the
       first message), so a session's subagents spread across accounts.
       `sessions.items[].id` in status is the composite key, load is counted
       per conversation, and a persisted concurrency cap re-learns
  #378 a `thread: continue` bound for a per-account third-party upstream is
       refused with the 400 Anthropic gives, so the client resends the whole
       conversation; `messageThreads: true` opts a relay out
  #434 a 429 whose x-codex-* headers show a spent account-wide window holds
       the account like an Anthropic rejection; a spent model-scoped bucket
       only moves the request
  #437 a Codex response head is awaited for five minutes (Anthropic unchanged)
  #411 idle keep-alive connections are held 120s on both listeners
  #389 with session distribution on, requests carrying no session id rotate on
       a cursor of their own instead of all resting on the current account
  #405 `defaultClientMode` ("mitm" | "base-url") sets what `run` and `env` do
       without a flag; `--mitm` / `--no-mitm` decide per launch, and in
       base-URL mode `env` unsets a stale proxy export naming this proxy
  #439 route `--bucket` is validated; an array `switchThreshold` reads as the
       default with one line saying so

Features
  #419 an MCP management endpoint at POST /teamclaude/mcp, off unless
       `proxy.mcp` is "read" or "full"; a named client key is read-only even
       in full mode, and with no proxy key it serves only this machine
  #428 per-account `switchThreshold`, a number or a per-bucket table
  #406 a Claude+Codex pool is drawn as two panes on a wide terminal, each with
       its own current marker; #392 names the provider in a mixed list; #418
       lets the operator arrange the list (`displayOrder`); #376 draws
       loopback-served accounts last; #435 shows the percentage beside a bar's
       countdown; #394 shows the running version in the header
  #430 free Codex rate-limit reset credits in status, the TUI and the dashboard
  #385 `proxy.terminalOnly` tunnels chatgpt.com so ChatGPT Desktop stays out

Fixes
  #404 a TUI paint can no longer block the proxy (stdout non-blocking, frames
       dropped while the terminal is behind); #410 a dead terminal no longer
       takes the proxy with it, and SIGHUP shuts down cleanly
  #433 token usage is booked from Codex Responses streams
  #386 #387 #388 the Codex five-hour window is read from the model-scoped
       family and the usage probe, and extra limits are named from their entries
  #431 a headerless 429 that follows the request is retried once
  #432 #439 startup and collaborator log lines reach the TUI's activity pane
       and log file instead of the covered terminal
  #415 #403 #439 reload mirrors `priority`, `disabled`, `stripRequestFields`
       onto the config entry, and a reload during a removal does not re-add it
  #439 the Host check uses the address actually bound; sx.org calls time out
  #381 the dashboard polls status before asking for a key
  #403 session outcome accounting classifies the decoded path; account names in
       daemon log lines are sanitised

Tooling
  #371 #372 #373 `npm run typecheck` (tsc over the JS sources) in CI, with a
       strict-mode ratchet: per-file strict diagnostics may not grow past the
       pre-merge commit (2006 at introduction, 1735 now)
  #401 docker workflow actions bumped

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
lifrary added a commit to lifrary/teamclaude that referenced this pull request Sep 29, 2026
Takes KarpelesLab/teamclaude up to ba01b4f while keeping the fork's routing
safeguards (403 cooldown without parking, send-failure fail-over, outbound
content-length, the single predispatch wait budget, cappedMessage, the
overload slot release, the dead-refresh-token guard and refresh retry, the
session home wait from d7810f1, and unranked-priority semantics).

Adopted from upstream, among others: per-conversation session pins (KarpelesLab#438),
401 fail-over without parking (KarpelesLab#439, KarpelesLab#473), real quota-reset retry-after and
candidate counts (KarpelesLab#429, KarpelesLab#408), a headerless 429 retry (KarpelesLab#431), fail-over on a
failing 200 stream (KarpelesLab#470), per-account egress proxies (KarpelesLab#441), extra-usage
fallback (KarpelesLab#427), maxSpend (KarpelesLab#466), stripOverageHeaders (KarpelesLab#478), keep-alive that
outlives the client pool (KarpelesLab#411), a dead terminal not killing the proxy
(KarpelesLab#410), console resolution per call (KarpelesLab#432), config reload and sync fixes
(KarpelesLab#465, KarpelesLab#415), and the dashboard, TUI and Codex work since 1.1.20.

Integration fixes the merge needed beyond conflict hunks:
- upstream request-path code that referenced upstream-only locals (sx,
  route, ctx.tried) rewritten for the fork's forwardRequest
- resolveSwitchThreshold was declared twice after a clean auto-merge
- the fork's warm-up probe now forwards a routed account through its own
  proxy instead of sending its credential direct (new regression test)
- a routing failure during a token refresh arms the routing hold instead of
  parking the account; a pinned request may use an account on routing hold
- the headerless-429 branch no longer writes after headers were sent
- canonical-state allowlists widened for upstream's new quota fields; saves
  use exportState()
- the fork's home wait keys on the conversation pin like selection does

Tests adapted where the fork deliberately differs (warm-up probe on by
default, account-anchored TUI cursor, coordinator-built Prober and Warmer,
unranked priority, per-conversation pin keys, fail-over-only 401, the wait
budget instead of inline waits), each with an in-file note. Full suite
2919/2919, lint, typecheck and the strict ratchet (1411 vs 1465) pass.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
rikbrown added a commit to rikbrown/teamclaude that referenced this pull request Oct 7, 2026
"Error: write EPIPE" on its own names neither the stream nor the call, which
is what made a broken pipe take three occurrences to pin down. Entries now
carry err.code and err.syscall beside the kind.

The first version of this also absorbed a broken pipe process-wide. Upstream
KarpelesLab#410 dropped that deliberately: it swallowed a broken pipe from any stream,
against the daemon's rule that an uncaught error is fatal, and the TUI's own
stdout/stdin listeners already cover the terminal case.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants