Registry marks live agents dead: one missed node heartbeat reaped 11 running processes in 1s, no grace period, no verification #1458

Summary

The broker's per-node agent set and the node's own activeAgents counter disagreed for two weeks with nothing reconciling them — and when the node lost heartbeat, the control plane swept 11 healthy, running processes to offline one second later, with no grace period and no verification.

Measured on finn-mini (node_d4190c4c2ca5c26bf547301347af4028, relay-broker/11.4.0) on 2026-08-07.

Part 1 — the counters drifted for a fortnight

• The broker had 20 agent records bound to that nodeId.
• The node self-reported activeAgents: 11.

The extra 9 were stale records from 2026-07-24 and 2026-08-02 — finn-pr1358-setup2/setup3/supervisor/direct-final/resume-node-final/fixed-supervisor, herdr-v1-codex-final-0724, finn-codex-signoff-c333b, closure-reviewer. They sat bound to a live node for two weeks. Nothing noticed, because nothing compares the two numbers.

Part 2 — the sweep declared live processes dead

finn-mini last heartbeat: 2026-08-07T11:52:35Z.
All 20 agent records transitioned to offline at 2026-08-07T11:52:36Z — exactly one second later, every one with release: null.

Those agents were not dead. On-host verification via os.kill(pid, 0) (not by reading a state file) confirmed all 11 alive as direct children of the broker, STAT S:

relaycast-306-liveness-regression-0806    35130   cloud-variant07-org-panel-0806          35183
daytona-fleet-node-liveness-0806          35377   cloud-platform-reliability-lead-0806    35726
cloud-execution-orchestration-lead-0806   35855   cloud-fleet-coordination-lead-0806      36011
cloud-identity-credential-lead-0806       36321   cloud-control-plane-tenancy-lead-0806   36462
chief-repo-backlog-cleanup-0806           37085   aihist-cloud-push-and-token-0806        44756
personal-agents-handoff-f1                57107

Their lstart values span 2026-08-06T19:24:35Z → 20:05:36Z, matching the registration window derived independently from the control plane.

The node process itself was also alive — PIDs running continuously since 2026-08-06T10:40:21Z, ~25h before the heartbeat loss, launchd never saw an exit, no reboot, no sleep, no crash report, no OOM. The registry was wrong about 11 running processes for 81 minutes, until an unrelated restart.

Why this matters

One missed heartbeat from one node is treated as proof of death for every agent attributed to it. There is no grace period, no re-probe, and no reconciliation against what the node actually reports. A transport hiccup is indistinguishable from a fleet-wide agent loss, and the registry commits the wrong answer to state rather than merely displaying it.

Suggested fixes

1. Reconcile the two counters. If the broker's per-node set and the node's activeAgents disagree, that is a bug worth surfacing rather than silently carrying for two weeks.
2. Do not sweep on a single missed heartbeat. A grace period, or a re-probe before transitioning agents, would have avoided all of this.
3. Distinguish "the node stopped talking to us" from "the node's agents died." They are different facts and only the first was observed.
4. Expire stale per-node bindings, so a fortnight-old record cannot be swept as though it were live.

Related: the same incident surfaced sweepStaleAgents having had no caller since the Fly.io→Cloudflare migration (relaycast#306) — 305 of 329 active records stale past TTL, oldest reading active for 23.9 days. These are opposite failures of the same missing primitive: nothing expires stale records, and a single missed beat expires live ones.