fix(v1): retire dead-tunnel servers from the elastic interception pool

[JannikSt]

The elastic pool never notices when a server's prime tunnel dies. The tunnel service reaps registrations whose frpc connection stayed down >60min (terminal status, frps rejects every reconnect), after which the server's URL 404s forever — but it stays in the pool with load 0 and keeps absorbing rollout assignments. Hit this in prod on a long GLM-4.5-Air SWE run: a network flap killed a third of the tunnels and ~80% of rollouts fast-failed on dead URLs for hours.

• PrimeTunnel.is_alive(): frpc alive + registration not in a terminal status. Status-based on purpose — a terminal registration still exists, so the SDK's check_registered existence probe stays True; transient API failures report alive so a flaky control plane can't mass-retire healthy servers.
• ElasticInterceptionPool health loop (60s): retires dead servers so new rollouts land elsewhere and the pool grows back with fresh tunnels; retired servers drain in-flight rollouts before teardown.
• Also retires zombies (terminal status but still routing on a surviving connection) proactively — they die for good on the next drop.

---

Note

Medium Risk
Changes rollout routing and tunnel lifecycle in the elastic pool; incorrect liveness logic could retire healthy servers or leave bad ones longer (up to the poll interval).

Overview
Fixes a case where dead prime tunnels stayed in the elastic pool with zero load and kept taking new rollouts on URLs that 404 forever.

Tunnel.is_alive() is added on the base tunnel (defaults to alive). PrimeTunnel tracks the active SDK client during expose() and implements liveness: frpc must be running, registration must exist, and status must not be expired/terminated. Transient tunnel API errors are treated as alive to avoid mass-retiring healthy servers.

ElasticInterceptionPool starts a 60s health loop when tunnels are required. Dead servers are removed from the active list into a draining queue (in-flight rollouts finish), then stopped once load hits zero. stop() now cancels both warm and health tasks.

^{Reviewed by Cursor Bugbot for commit 28809de. Bugbot is set up for automated code reviews on this repo. Configure here.}

[macroscopeapp]

Retire dead-tunnel servers from the ElasticInterceptionPool active pool

• Adds a background _health_loop task to ElasticInterceptionPool that periodically checks tunnel liveness every 60 seconds when requires_tunnel is True.
• Servers whose tunnels are dead are moved to a _draining list and removed from the active pool, stopping them fully once in-flight rollouts reach zero load.
• Adds Tunnel.is_alive() with a default of True; PrimeTunnel overrides it to return False when frpc is not running, the registration is missing, or the registration status is expired/terminated. Transient control-plane errors report alive to avoid false retirements.
• ElasticInterceptionPool.stop now cancels and awaits both the warm task and the new health task.
• Behavioral Change: tunnel-backed servers that lose their registration are no longer eligible for new rollouts and will be stopped once drained.

^{Macroscope summarized 28809de.}

[JannikSt]

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. More of your lovely PRs please.

Reviewed commit: 28809de410

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[macroscopeapp]

Approvability

Verdict: Needs human review

Unable to check for correctness in 28809de. This PR introduces a new background health-checking task and server retirement logic that fundamentally changes how the elastic interception pool manages its servers at runtime. While addressing a legitimate issue, these significant runtime behavior changes warrant human review.

^{You can customize Macroscope's approvability policy. Learn more.}