[Bug]: T3 Connect never recovers a reaped tunnel: cloudflared's rejection is invisible over QUIC and reads Tunnel not found over HTTP/2 #16399

Activity

1. juliusmarminge commented on Oct 6, 2026

   

   juliusmarminge

   on Oct 6, 2026

   Member

   More actions

   Note

   Grok responding on behalf of Julius.

   Triage

   Thanks for the cloudflared traces and the matcher write-up, @berghtho. They made this much easier to follow. This looks like its own recovery gap, separate from #16258.

   What I found

   • A reaped tunnel with an unchanged origin never asks Cloudflare. reconcileOrigin in infra/relay/src/environments/ManagedEndpointProvider.ts returns ready when the stored host and port still match, so startup registration keeps the old token. That matches docs/internals/t3-connect.md: registration only touches Cloudflare when the origin changed.
   • Replacement happens only if the child exits (superviseConnector queues a recovery request) or isRejectedRelayClientTunnelOutput matches four times. The matcher in apps/server/src/cloud/ManagedEndpointRuntime.ts requires Register tunnel error from server side plus one of Failed to get tunnel, Record for tunnel not found, or Invalid tunnel secret. Killing the process works because the exit path doesn't consult the matcher; restarting the app doesn't, because the same stored token is started again.
   • On the pinned cloudflared 2026.5.2, the QUIC path can't hit that matcher. quicConnection.Serve returns &ControlStreamError{} (control stream encountered a failure while serving), and the supervisor only prints Register tunnel error from server side for ServerRegisterTunnelError, so this falls through to Serve tunnel error. The managed child is started without --protocol, so it stays on QUIC. Your debug run matches that source.
   • HTTP/2 does print Register tunnel error from server side, but Unauthorized: Tunnel not found fails the reason regex. So forcing HTTP/2 (T3 Connect: let users force the managed cloudflared to HTTP/2 — on QUIC-blocking corporate networks the tunnel stays down because cloudflared won't fall back #16344) alone wouldn't make the matcher fire.
   • [Bug]: T3 Connect stays down after a network change: cloudflared keeps running with zero connections and is never restarted #16258 is a live connector with no route to the edge (/ready stays 503 without a registration rejection), while this is a deleted tunnel whose rejection text isn't recognized. [Bug]: T3 Connect treats a spawned but unreachable tunnel as 'running' #7447 and [Bug]: Desktop-hosted T3 Connect keeps no record of the managed cloudflared's output #16331 are the visibility problems you noted, not this recovery miss.

   Likely fix area

   Extending the matcher is one narrow option, with a few details worth keeping in mind:

   • On HTTP/2, Tunnel not found could join the existing reasons, still requiring Register tunnel error from server side.
   • On QUIC, counting only Serve tunnel error with control stream encountered a failure while serving (not the failed to serve tunnel connection line that repeats it for the same attempt) would avoid double counting. ControlStreamError is also what a graceful full-process unregister returns, so the existing four-failure threshold and two-minute cooldown would still matter.
   • One open question: every unmatched ERR line is normally logged as Relay client reported a transport warning, and that message is missing from your trace. The --loglevel debug hand-run shows what cloudflared prints, but not that the managed --loglevel info child's stderr actually reached observeConnectorOutput. A matcher change would only recover the host if those lines are delivered, so that path may need a look too.

   A maintainer will decide on the fix direction.
2. juliusmarminge

   added

   bugSomething is broken or behaving incorrectly.Something is broken or behaving incorrectly.

   via-triageFiled through npx t3 triageFiled through npx t3 triage

   on Oct 6, 2026
3. bitflippr

   added a commit that references this issue on Oct 6, 2026

   fix(connect): recover a reaped tunnel the connector cannot register

   ...

   This issue will close once commit 34e8ddf is merged into the 'main' branch.

   34e8ddf
4. Heggtor commented on Oct 6, 2026

   

   Heggtor

   on Oct 6, 2026

   More actions

   Also observed on macOS (Apple Silicon), T3 Code nightly 0.0.46-nightly.20261005, with the managed cloudflared 2026.5.2, connecting from the iPhone TestFlight app.

   Live Activities / Dynamic Island and push notifications continued working, but opening a thread failed with endpoint_request_failed.

   During diagnosis, cloudflared returned Unauthorized: Tunnel not found, and automatic recovery did not trigger. Restarting the desktop app had reused the same unavailable tunnel.

   Terminating the managed cloudflared process caused T3 to provision a replacement within about 4 seconds. The replacement registered 4 connections, and an HTTP request to its public endpoint returned 200 and increased the tunnel request counter.

   After this workaround, I confirmed that the iPhone app connects successfully and I can open my chats again.
5. stilak12 commented on Oct 6, 2026

   

   stilak12

   on Oct 6, 2026

   More actions

   Confirmed on macOS (Apple Silicon) + Windows 11, both on 0.0.46-nightly.20261005.2702, pinned cloudflared 2026.5.2, desktop app as host on both machines.

   Additional trigger — releaseManagedTunnelOnShutdown makes this hit on every clean quit, not just reaped-idle tunnels. The server deletes the provisioned Cloudflare tunnel on shutdown (unless an update restart is pending), but cloud-endpoint-runtime-config with the old tunnelId/token survives. On the next launch, the recovery path registers the stored tunnelId — reports ready without touching Cloudflare — and the managed cloudflared comes up with a dead token:

   cloudflared_tunnel_tunnel_register_fail{error="server_error",rpcName="registerConnection"} = 14 and climbing
   cloudflared_tunnel_ha_connections 1
   cloudflared_tunnel_total_requests 0
   

   Relay then reports environment_endpoint_unavailable / endpoint_request_failed to every client ("Reconnecting: Relay could not reach the environment endpoint").

   Observed sequence on macOS:

   1. t3 connect unlink + t3 connect + app restart → fresh provision, tunnel healthy (ha_connections 4, requests flowing), remote clients connect fine.
   2. Clean quit → relaunch → new cloudflared spawned with the same runtime config (cloud-endpoint-runtime-config mtime unchanged from the previous provision; cloud-endpoint-confirmed-origin rewritten this startup) → registerConnection server_error loop, identical to the reaped-tunnel case. reconcileDesiredLinkIfStillDesired either didn't run or didn't reach applyCloudRelayConfig — the config file was never rewritten.

   So the release-on-shutdown + stale-runtime-config path makes this deterministic per clean relaunch, not only after hours of downtime. Symmetric failure on both machines: each showed endpoint_request_failed for the other until its own link was re-provisioned.

   Workaround that worked here: npx t3@nightly connect unlink → npx t3@nightly connect → full app restart on the affected host (re-provisions a fresh tunnel under the same environment). The lighter kill cloudflared workaround from this issue also holds.
6. Hazzajenko commented on Oct 6, 2026

   

   Hazzajenko

   on Oct 6, 2026

   More actions

   Also seen on Windows 11 Pro (10.0.26200, x64). The host is the desktop app 0.0.45-nightly.20261002.2572 with the pinned cloudflared 2026.5.2. The clients are the desktop app on a second Windows laptop and the Android app 1.4.0. So the bug is present from the 2 October nightly at the latest.

   The trigger was a clean OS restart, the same path that @stilak12 describes. The host server ran without a stop for 14 hours. The last good request from a client through the relay was at 01:55 local time. The OS restarted at 03:37. The app started again at 09:30. After that, both clients showed Reconnecting: Relay could not reach the environment endpoint (endpoint_request_failed).

   State 10 minutes after startup:

   GET 127.0.0.1:20241/ready -> 503 {"readyConnections":0}
   cloudflared_tunnel_tunnel_register_fail{error="server_error",rpcName="registerConnection"} 55
   cloudflared_rpc_client_latency_secs_bucket{...register_connection,le="0.05"} 55
   cloudflared_tunnel_ha_connections 1
   cloudflared_tunnel_total_requests 0
   GET https://<endpoint>.t3coderelay.com/.well-known/t3/environment -> 530
   

   Each registration failed in less than 50 ms. The server trace has no recovery span after the startup activateManagedTunnel, so the four-failure matcher never fired.

   Two more details:

   • The managed cloudflared output is not on disk anywhere. The desktop app writes backend child output to server-child.log only when the child fails, and that file was last written 5 days earlier. The trace file has no Relay client reported a transport warning entries either. This agrees with the open question in the triage comment about whether those lines reach observeConnectorOutput. On this host, it was not possible to see the rejection text without a hand run.
   • The workaround from this issue worked. I stopped the managed cloudflared process, and within about 10 seconds a replacement started. /ready then reported readyConnections: 4 and the public endpoint returned 200. The pairing and the hostname stayed the same.

   Found during a t3 triage session by Claude Code with Claude Opus 5.5.
7. mwolson commented on Oct 7, 2026

   

   mwolson

   on Oct 7, 2026

   Contributor

   More actions

   Adding a data point on why the stored runtime config survives a clean quit, which @stilak12 described above. The desktop app gives the backend 2 seconds after SIGTERM before it sends SIGKILL (DEFAULT_BACKEND_TERMINATE_GRACE in apps/desktop/src/backend/DesktopBackendManager.ts), while the server allows releaseManagedTunnelOnShutdown 10 seconds (apps/server/src/server.ts). Release stops the connector, sends the relay DELETE, and removes cloud-endpoint-runtime-config only after an ok: true response.

   Traces from five quits on one macOS host (Nightly from Oct 6, pinned cloudflared):

   • Four releases finished in 1.37 to 1.79 seconds, 0.2 to 0.6 seconds before the SIGKILL. Each next launch ran the full link reconcile and connected.
   • In the fifth, release stopped the connector within 3 ms, and the server was SIGKILLed 2.04 seconds after quit began, with the release span never completed. The next launch took the recovery path with the stored tunnel, registration came back ready, and cloudflared sat at zero connections for 46 minutes until I killed it. Recovery then handed back a different tunnel ID.

   So whenever the DELETE round trip runs past about 2 seconds, the relay most likely deletes the tunnel while the local config survives, and the next start reuses it. #16402 would let the host recover from that, but the race seems worth closing too, for example with a backend terminate grace longer than the release budget, or by marking the stored config stale before sending the DELETE so the next start reconciles.

   ---

   Sent by Mike's agent (Claude Opus 5.5 in T3 Code)
8. mwolson commented on Oct 7, 2026

   

   mwolson

   on Oct 7, 2026

   Contributor

   More actions

   Follow-up: #16649 closes this race on the relay side (registration now checks
   Cloudflare, so the next start gets a replacement right away), and #16648 adds
   the three-minute readiness fallback, so the fix suggestions at the end of my
   comment above are moot. The traces there match the race #16649 describes.

   ---

   Sent by Mike's agent (Claude Opus 5.5 in T3 Code)
9. juliusmarminge

   closed this as completedin #16648on Oct 7, 2026