Cross-node attach 404s node-wide: broker heartbeat and inventory.sync disagree on which agents are running (roster drift; 9/10 agents on sf-mini unattachable)

[khaliqgant]

Summary

Cross-node terminal attach fails with HTTP 404 agent_not_found for 9 of the 10 agents running on sf-mini, and for 1 of 9 on finn-mini. The node is healthy, the agents are alive, and the node truthfully advertises them every 12s.

The failure is not in the node's terminal subsystem. It is a divergence between two independent in-process sources of truth inside the same broker for "which agents am I running", only one of which the control plane uses to authorize attach.

This is not an sf-mini-specific outage. sf-mini is the same fleet-wide defect at 90% instead of 11%.

Measured drift (2026-08-23 ~10:40Z)

node	ver	heartbeat active_agents	relay:live-agents:v1 names	roster rows active on node	drift
sf-mini	11.8.1	10	10	1	9
finn-mini	11.8.1	9	9	8	1
cloud-3129-sched-code	11.8.2	1	1	1	0

Every agent in the "drift" column is unattachable. Every one not in it attaches fine.

Reproduction (with control)

$ agent-relay node agent attach sandbox-lead-0822 --node sf-mini --mode view
Error: The terminal-session request was rejected. Upstream message
"No active agent named 'sandbox-lead-0822' on node 'sf-mini'".
endpoint "https://cast.agentrelay.com/v1/nodes/sf-mini/terminal/sessions";
HTTP 404; code agent_not_found; attempts 1 (not retried because the failure was terminal).

$ agent-relay node agent attach branch-triage-0822 --node finn-mini --mode view
{"kind":"worker_stream","name":"branch-triage-0822",...}   # succeeds, live screen

agent-relay fleet agent list already labels the defect — it is the only surface that shows both sides:

sf-mini    sandbox-lead-0822       remote live          <- node says live, roster does not bind it
finn-mini  branch-triage-0822      remote live+roster   <- both agree

Root cause

1. Attach authorizes off the roster, which the node cannot repair.

cloud/packages/relaycast/src/fleet/routes.ts:355 gates session creation on a five-predicate row lookup:

SELECT id FROM agents
WHERE workspace_id = ? AND name = ? AND status = 'active'
  AND location_type = 'via_node' AND location_node_id = ? LIMIT 1

Node liveness (findTerminalAttachNode, isLive) passes before this, and the terminal-transport check (terminal_connected) runs after it — so a node with a perfectly healthy terminal transport never gets that far. Two different predicates fail in the wild, both surfacing as the identical 404:

• status = 'active' — 9/10 sf-mini agents are offline in the roster (sandbox-lead-0822: last_seen 2026-08-23T08:53:14Z, frozen ~1h45m).
• location_node_id = ? — abx-protocol-review is roster-active but bound to cloud-3061-repro-0817, while both sf-mini and finn-mini advertise it live. Attach on sf-mini 404s for this one too, via the other predicate.

2. The two sources of truth.

In relay/crates/broker/src/node_control.rs the broker maintains two independent sets:

• load.active_agent_names → rebuilt into the relay:live-agents:v1 capability on every heartbeat, 12s (heartbeat(), ~line 503).
• inventory → sent as inventory.sync on connect, on UpdateInventory, and on a 60s inventory_refresh tick (INVENTORY_REFRESH_INTERVAL, line 34).

Only inventory.sync writes the roster. reconcileNodeInventory (cloud/.../durable-objects/node.ts:330) sets status='active' for names in the payload, and at line 359 actively sweeps to offline any roster-active agent bound to the node that is absent from the payload.

3. Why it is permanent, not transient.

The reclaim guard admits any agent that is not currently active:

AND ( location_node_id = ?  OR  status != 'active'  OR  (location_type='via_node' AND location_node_id IS NULL) )

sandbox-lead-0822 is offline, so status != 'active' is TRUE — the very next sync should reclaim it. sf-mini has sent ~100 inventory.syncs since 08:53 and it is still offline.

⟹ sf-mini's inventory vector does not contain these agents, while its active_agent_names does. The sync is running and omitting them. Something (a reconnect, or a broker restart that re-adopted the PTYs) repopulated active_agent_names without repopulating inventory; the 08:53 sweep then marked all of them offline and every 60s sync since has re-confirmed their absence.

relay:live-agents:v1 — the one signal that is correct — has zero consumers in the control plane. It exists only in relay/packages/cli/src/cli/lib/fleet-live-agents.ts. grep across cloud/packages/relaycast returns nothing. The node can shout the truth every 12s forever and attach will never hear it.

Why this also explains #1593

#1593 reports DMs silently dropped for agents alive 1h15m–5h, every send returning recipientMatched: true. That is the same roster row. Delivery routes on location_*/status; attach authorizes on location_*/status. One stale row breaks both, which is why the symptoms share an age threshold and why pending_messages is 0 — the route is resolved against a row that no longer points anywhere live. I believe these are one bug, not two.

Proposed fix

Immediate (control plane, unblocks 10 stranded agents): treat relay:live-agents:v1 as authoritative for attach. In handleCreateTerminalSession, when the roster lookup misses, fall back to the node's heartbeat live-agent set before returning 404 — the node is the ground truth for its own processes. Attach is already gated on a workspace key and node liveness; this adds no authorization surface.

Correct (broker): collapse the two sets. active_agent_names and inventory should be one structure, or inventory should be derived from the live PTY set at send time rather than accumulated by command. As long as they are independently maintained they will diverge again.

Defensive (control plane): the line-359 sweep should not mark an agent offline while the same node's heartbeat still advertises it as live. Right now inventory.sync can un-person an agent that the heartbeat says is running.

Observability: fleet agent list already computes the drift. Nothing alerts on it. A node whose live_names − roster_active > 0 is silently accumulating unmanageable agents.

Recovery of the 10 stranded agents — needs a decision

UpdateInventory triggers an immediate resync, and spawning any agent on sf-mini issues one. If the broker's inventory vector still holds the 10 agents, a spawn would reclaim all of them at once and restore attach. If it does not, they stay offline and we have learned the vector is genuinely empty.

That is a cheap, non-destructive experiment (spawn, not restart, not release) — but it mutates a node holding 10 agents with possibly-unpushed work, so I am not running it without a go-ahead. The alternative is a control-plane row correction, which is a write to agents and also needs sign-off.

Environment

• Observed from chief-broker (relay-broker/11.8.0), CLI 11.8.1, gh khaliqgant.
• Note: chief-broker at 11.8.0 does not publish relay:live-agents:v1 at all — nor do the 11.6.9 nodes. fleet agent list degrades those to count only (degraded) / "inventory unavailable", so drift there is not even measurable.

[khaliqgant]

Correction and re-verification

Two things in the original report need fixing, and a third — a proposed cheap diagnostic — turns out to be unsound. The core root cause is unchanged and now has more evidence behind it.

1. Methodology error: I measured the wrong column

My drift table joined on agents.metadata.fleet.node_id. That field is registration provenance, written once at agent.register. The terminal lookup uses location_node_id, which reconcileNodeInventory rewrites on every claim. They diverge as soon as an agent is re-claimed by another node.

Consequence: finn-mini's reported drift of 1 was an artifact. finn-mini is genuinely 0 drift. The corrected table:

node	live names	attachable	real drift
sf-mini	11	1	10
finn-mini	10	10	0
cloud-3129-sched-code	2	2	0

2. Failure mode 2 named the wrong node

I wrote that abx-protocol-review is bound to cloud-3061-repro-0817. That was metadata.fleet.node_id again. Its current binding is finn-mini:

attach abx-protocol-review --node finn-mini  -> worker_stream, worker_stream, worker_stream   (3/3 OK)
attach abx-protocol-review --node sf-mini    -> agent_not_found ×3                            (3/3 404)

The mechanism I described is unchanged and now directly demonstrated: an agent that is roster-active and advertised live by a node still 404s there when location_node_id points at a different node. sf-mini is advertising an agent that has since migrated to finn-mini. I had the right mechanism and the wrong node.

3. The PRESENCE column is a useful screen but is NOT a sound drift diagnostic

It was suggested that fleet agent list --pretty's remote live+roster vs remote live gives a free fleet health check. It is a good first screen, but it over-reports attachability, because +roster means "this name is active in the roster somewhere" — it does not verify the binding is to this node:

$ agent-relay fleet agent list --pretty
finn-mini  abx-protocol-review   remote live+roster
sf-mini    abx-protocol-review   remote live+roster     <- says attachable
$ agent-relay node agent attach abx-protocol-review --node sf-mini
agent_not_found ×3

So a dual-advertised name reads as healthy on both nodes while being attachable on only one. Anyone using this as a health check will under-count drift by exactly the second failure mode. It is worth shipping as a screen with that caveat stated, not as the measurement.

The sound check is attach itself, or a roster query on location_node_id rather than the metadata field.

4. Re-verification of the original claims

A flake mode was reported — a single agent_not_found against a healthy agent that then attached cleanly on retry. I could not reproduce it in 12 attempts across four spaced triples. Every result was 3/3 consistent in both directions:

target	node	result
sandbox-lead-0822	sf-mini	404 ×3
abx-protocol-review	sf-mini	404 ×3
abx-protocol-review	finn-mini	attached ×3
branch-triage-0822	finn-mini	attached ×3

That does not disprove the flake — it may be load- or lock-dependent, and a single 404 should still be retried before being recorded. But the sf-mini 404s in this report are not flake artifacts.

5. The reclaim experiment ran, and it confirms the diagnosis

A spawn on sf-mini (sandbox-lead-0823, 10:38:46Z) forced the UpdateInventory → inventory.sync path. Nine minutes and ~9 syncs later the stranded agents were still agent_not_found, while the newly spawned agent attached normally.

This is the predicted result and it closes the question: sf-mini's inventory vector genuinely does not contain the stranded agents. The sync is running and omitting them. A forced resync is not a recovery path — the remaining options are a control-plane row correction or salvage-and-respawn.

[kjgbot]

sf-mini salvage probe — Step 1 node-local enumeration

Source: AGENT_RELAY_STATE_DIR=~/.agentworkforce/relay/sf-mini-node/state agent-relay node agent list --status, run locally on SF-Mac-Mini at 2026-08-23 21:37 Europe/Oslo. No broker was started or restarted.

requested agent	current_state	last_activity_at (UTC)	pid	sessionId
sandbox-lead-0822	idle	2026-08-23T08:25:30.220762042Z	63872	e5bb5ccd-191c-4ee7-b66e-f58a73e0941e
sandbox-lead-0823	idle	2026-08-23T12:15:48.972734250Z	47933	4aaeb517-37cd-43ac-92ac-04a2eae4b75f
sessions-turns-backend-0822	idle	2026-08-22T21:23:30.518330291Z	30513	888b0f30-7b62-4bc1-86c2-26d85b91b056
agentbox-transcript-0822	idle	2026-08-22T23:05:00.141115459Z	35698	a62ec769-8f73-431c-a9f3-8c4e53ee1f98
factory-placement-why-0823	absent from node-local table	—	—	—
abx-prewarm-oc	working	2026-08-23T19:37:06.921338084Z	20557	null
abx-prewarm-review	idle	2026-08-23T16:19:29.123015959Z	20585	0b92dfb1-8f42-4661-90ce-76a78c6f95a4
abx-protocol-oc	idle	2026-08-23T09:06:34.577971250Z	20280	null
abx-seams-oc	idle	2026-08-23T16:20:57.709985Z	20673	null
abx-seams-review	idle	2026-08-23T17:34:58.291858417Z	20707	e1bc37d0-1432-4406-bdd3-5f8fba545bca

The local table also contains abx-protocol-review (idle, pid 20366, session b294e9e4-7324-483e-8aec-079a742b4de1) and this probe; neither is in the requested ten-agent salvage set.

Proceeding agent-by-agent with cwd/worktree/HEAD-remote/dirty-state checks and immediate safety pushes where needed.

[kjgbot]

sf-mini salvage probe — Step 2 batch 1 (filesystem + immediate safety pushes)

All cwd checks used /usr/sbin/lsof -a -p <pid> -d cwd -Fn.

Deleted under live processes

agent	pid	lsof cwd	filesystem result	provisional grade
sandbox-lead-0822	63872	/Users/khaliqgant/Projects/AgentWorkforce/.lanes/sandbox-lead	path no longer exists	worktree gone
sessions-turns-backend-0822	30513	/Users/khaliqgant/Projects/AgentWorkforce/.lanes/sessions-turns	path no longer exists	worktree gone

The processes are alive, but their cwd paths have been deleted. Branch/HEAD/status cannot be queried through those missing paths. I am continuing through surviving git worktree metadata and remote refs to recover their last known branch/HEAD.

Seven agents share the surviving chief checkout

Agents: sandbox-lead-0823, agentbox-transcript-0822, abx-prewarm-oc, abx-prewarm-review, abx-protocol-oc, abx-seams-oc, abx-seams-review.

• cwd/worktree: /Users/khaliqgant/Projects/AgentWorkforce/chief (exists)
• branch: agent/chief-broker-spawn-grok
• HEAD: 5ed681d8c85e71b55da05191bfdce665377b51d7
• after git fetch origin --prune, git branch -r --contains 5ed681d... returned no refs; exact-tip git ls-remote origin also returned none
• kjgbot GitHub permission on AgentWorkforce/chief: push:false (verified via API)
• uncommitted state in the shared checkout:
  • ?? .briefs/
  • ?? .reports/
  • ?? principals/khaliq/workstreams/sandbox-router.md

Because chief is read-only to this identity, I immediately preserved the unpushed commit object in the writable AgentWorkforce/sandbox repo. Created:

• safety/sandbox-lead-0823-5ed681d8c85e
• safety/agentbox-transcript-0822-5ed681d8c85e
• safety/abx-prewarm-oc-5ed681d8c85e
• safety/abx-prewarm-review-5ed681d8c85e
• safety/abx-protocol-oc-5ed681d8c85e
• safety/abx-seams-oc-5ed681d8c85e
• safety/abx-seams-review-5ed681d8c85e

All seven pushes succeeded. The three untracked paths are not committed and therefore are not on those safety branches; they are shared-checkout state and cannot safely be attributed to a particular stranded agent. No stash, commit, kill, release, restart, merge, or deploy was performed.

Provisional grade for these seven: holding uncommitted work in a shared checkout (the previously unpushed HEAD is now safety-pushed).

[kjgbot]

sf-mini salvage probe — Step 2 batch 2 (deleted-worktree reconstruction)

I used the stranded Claude session transcripts only to reconstruct git commands/results after the cwd paths vanished, then verified surviving refs/worktrees directly.

sandbox-lead-0822

• original repo (from its own recorded .git output): AgentWorkforce/sandbox
• missing cwd: .lanes/sandbox-lead
• the agent reported the path was deleted twice in-session
• last reconstructed cwd HEAD before the second deletion: detached 7c8d0ac, clean, recreated from origin/main
• current fetched sandbox refs containing 7c8d0ac: origin/main (and several descendants); therefore that detached cwd HEAD was already remote
• crucially, the transcript identified a surviving auxiliary worktree at /private/tmp/chief-sandbox-workstream used for its handoff document commits:
  • branch lane/sandbox-router-workstream-0822
  • HEAD 83929183f5d980245084961a68e702c4ecc2b21f
  • status clean
  • neither fetched AgentWorkforce/chief remote refs nor exact-tip git ls-remote origin contained this HEAD
  • kjgbot cannot push to chief

Immediate salvage completed: pushed the full five-commit handoff chain to writable AgentWorkforce/sandbox branch safety/sandbox-lead-0822-83929183f5d9.

Grade: worktree gone, with its unpushed auxiliary commit chain now safety-pushed.

sessions-turns-backend-0822

• missing cwd: .lanes/sessions-turns
• original repo: AgentWorkforce/relay
• last recorded direct audit (2026-08-22T19:58Z):
  • detached HEAD 2f24c6af2d1f017425db7ae54433dfedcac59b38
  • nothing to commit, working tree clean
• fresh verification against AgentWorkforce/relay:
  • git branch -r --contains 2f24c6af2 includes origin/main
  • exact git ls-remote origin shows 2f24c6af... at refs/heads/main

Grade: worktree gone; last known tree was clean and HEAD is pushed on origin/main.

No stash, kill, release, restart, merge, or deploy was performed.

[kjgbot]

sf-mini salvage probe — Step 2 batch 3 (factory-placement-why-0823)

The agent is absent from the current node-local PTY table and no matching process exists now.

Local historical evidence recovers:

• PTY wrapper pid: 44038
• Claude pid: 44057
• sessionId: 7f01b9db-46c3-4be0-a521-2b8830083932
• started: 2026-08-23 11:59 Europe/Oslo
• transcript ended: 2026-08-23T13:13:07Z
• transcript cwd: /Users/khaliqgant/Projects/AgentWorkforce/chief
• transcript branch metadata: agent/chief-broker-spawn-grok

Because the process is gone, lsof -a -p 44057 -d cwd -Fn cannot return a live cwd. The transcript's tool inventory contains only read-only shell/file reads and Relay messages: no Edit/Write tools, no git commit/push, and file-history snapshots contain no tracked-file backups. Its only scratch write was an extracted npm tarball under /tmp/factory-diag, outside the repo.

The shared chief checkout remains at unremote HEAD 5ed681d8c85e71b55da05191bfdce665377b51d7; I preserved that commit under this agent's name as AgentWorkforce/sandbox:safety/factory-placement-why-0823-5ed681d8c85e (push succeeded). Shared checkout dirty paths remain the three already reported in batch 1.

Grade: dead process. No agent-authored tracked repo mutation was found; its cwd commit is now safety-pushed.

No kill, release, restart, merge, deploy, stash, or credential action was performed.

[kjgbot]

sf-mini salvage probe — Step 2 batch 4 (secondary worktrees + uncommitted snapshot)

agentbox-transcript-0822

Although its process cwd is the surviving shared chief checkout, its transcript shows all authored code work occurred in /Users/khaliqgant/Projects/AgentWorkforce/cloud-worktrees/agentbox-transcript, which is now gone.

Recovered and verified from the session record plus remote:

• branch: lane/agentbox-transcript-0822
• first commit: c8e4a81b (915 inserted lines)
• final fix commit: 1ae2556be6b91a2ec0c023cfa7ee8c0b30e381cc
• final push used --force-with-lease and succeeded
• fresh git ls-remote AgentWorkforce/cloud refs/heads/lane/agentbox-transcript-0822 returns exact tip 1ae2556be6b91a2ec0c023cfa7ee8c0b30e381cc
• the final commit records 28 green tests

Grade: finished and pushed. Its authored worktree was subsequently removed, but the exact final HEAD is remote.

Shared untracked-state resolution

Path-only secret scan across the three untracked locations reported in batch 1 found no credential/private-key patterns. The .briefs/ files date Aug 18–21 and .reports/ dates Aug 20, predating these stranded sessions.

The current principals/khaliq/workstreams/sandbox-router.md differs from the previously safety-pushed sandbox-lead-0822 five-commit version. Transcript evidence attributes the later edits to sandbox-lead-0823; that agent had already preserved them in gist kjgbot/2616e234aa4264edbbf142f379ed70cf.

I additionally created an immutable git snapshot without modifying the shared checkout and pushed it to:

• AgentWorkforce/sandbox:safety/sandbox-lead-0823-b56950a1adff
• commit b56950a1adffde4ae2b7a663ee82648244ceeef7

That commit overlays the exact current workstream file on the shared chief HEAD. Thus the only stranded-session-authored uncommitted file has now been git-safety-pushed. No secret value was read or posted.

[kjgbot]

sf-mini salvage probe — FINAL grades and recycle decision input

Finalized at 2026-08-23T19:49Z after node-local enumeration, cwd inspection, remote-containment checks, session reconstruction, safety pushes, temp-artifact preservation, and read-only TUI views. The broker was never started/restarted and no agent was killed/released.

agent	final grade	recycle evidence
sandbox-lead-0822	worktree gone	live process cwd deleted; last detached sandbox HEAD 7c8d0ac is on remote; five unpushed chief handoff commits salvaged to safety/sandbox-lead-0822-83929183f5d9
sandbox-lead-0823	finished and pushed	shared chief cwd exists; base HEAD safety-pushed; exact later workstream snapshot pushed as safety/sandbox-lead-0823-b56950a1adff (also present in its prior gist)
sessions-turns-backend-0822	worktree gone	live process cwd deleted; last audit clean/detached at 2f24c6af..., exact HEAD on AgentWorkforce/relay:main
agentbox-transcript-0822	finished and pushed	authored cloud worktree later disappeared, but exact final HEAD 1ae2556be6b91a2ec0c023cfa7ee8c0b30e381cc is remote tip of AgentWorkforce/cloud:lane/agentbox-transcript-0822
factory-placement-why-0823	dead process	historical pids/session recovered; no tracked writes found; shared cwd HEAD preserved as safety/factory-placement-why-0823-5ed681d8c85e
abx-prewarm-oc	finished and pushed	read-only investigation produced BLOCKED: report; read-only TUI view shows analysis complete and stuck only attempting the required Relay notification; report preserved under safety/abx-prewarm-oc-e31a5c9e39f5
abx-prewarm-review	finished and pushed	reviewer artifact preserved under safety/abx-prewarm-review-e31a5c9e39f5
abx-protocol-oc	finished and pushed	completed protocol report including independent REVIEW section preserved under safety/abx-protocol-oc-e31a5c9e39f5
abx-seams-oc	holding unpushed/incomplete work	required /tmp/ws-failure-seams-status.md never appeared; read-only TUI view shows it diverted into an unrelated Relay-token investigation and waiting for direction. No authored report could be recovered. The independent review substitute is preserved under safety/abx-seams-oc-e31a5c9e39f5, but this is not the missing OC deliverable.
abx-seams-review	finished and pushed	14 KB independent review substitute preserved under safety/abx-seams-review-e31a5c9e39f5

The common ABX safety commit is e31a5c9e39f5895207537f02b534f35d4fd9ed6b and contains all three available status artifacts. No credential/private-key patterns were detected in those reports or the shared untracked paths.

Recycle recommendation input

• Evidence-safe to recycle: sandbox-lead-0822, sandbox-lead-0823, sessions-turns-backend-0822, agentbox-transcript-0822, factory-placement-why-0823 (already dead), abx-prewarm-oc, abx-prewarm-review, abx-protocol-oc, abx-seams-review.
• Hold if the missing primary seam audit still matters: abx-seams-oc. Its expected deliverable does not exist and its current TUI is off-task.

The shared chief checkout still reports the old pre-session .briefs/ and .reports/ paths plus the workstream file, but the stranded-session-authored workstream content is now on a safety branch. I made no recycle action; this is decision evidence only.

[khaliqgant]

sandbox-substrate-lead-0823 — Step 0 environment report + interim re-verification

Picking up sandbox substrate ownership from sandbox-lead-0822 (stranded on sf-mini). Reporting early per instruction; the attach sweep behind §3 is still running.

1. Environment

host	Khaliqs-MBP.home (user khaliqgant)
pwd	/Users/khaliqgant/Projects/AgentWorkforce/sandbox-router
gh account	khaliqgant — repo, workflow, read:org, gist, delete_repo, admin:public_key. Push works, so deliverables from this box are not stranded the way the finn-mini report was.
node / npm	v22.22.2 / 10.9.7
disk	44 GiB free (data volume 90% used)
agent-relay	11.8.1 (11.8.2 available; not upgrading — no approval)
load	20.24 / 26.53 / 22.30 on a 4-day-uptime box — heavily loaded

Can anyone attach to me? No. agent-relay node status → STOPPED; fleet status → node.available: false, "broker did not report a node name". This machine publishes no fleet node, so I am reachable only via DM and via whatever drives this session. I have not started a broker — that would register a new fleet node and I don't have a ruling for that. Say the word and I'll bring one up.

Checkout staleness confirmed, and all three are on feature branches, so I am reading via git show origin/main:<path> and not pulling anything:

• sandbox → fix/scale-multi-root-flush-timeout-0823
• sandbox-router → agent/process-manifest-0820, 6 behind
• cloud → integrations/stripe-mcp-provider
• relay → fix/attach-fleet-hint-node-placement, 5 behind

2. Three corrections to the brief I inherited

(a) finn-mini is offline. The node designated as the working control for the comparison is node_216335751140745216, status: offline, and it appears only under fleet nodes --all history. The live fleet is sf-mini, chief-broker, and five cloud/sandbox nodes. I used cloud-3129-sched-code as the control instead.

(b) sf-mini is not "broken node-wide", and the errors are not byte-identical. Re-probing three agents today returns three different results, not one:

agent	result
sandbox-lead-0822	HTTP 404 agent_not_found, node id never resolved
factory-lead-sf-0824	passes the agent lookup, resolves node_203549044126121984, then fails at transport: snapshot HTTP 503, close 1011 terminal transport connected but did not become ready
sfmini-salvage-probe-0823	same as above — passes the lookup, fails at transport

That is two independent faults wearing one symptom, and the second one is invisible if you only test stranded agents.

(c) sf-mini the node is healthy. It heartbeats, is live: true, and advertises all 17 agents via relay:live-agents:v1. Nothing here justifies restarting or releasing anything on it — and doing so would destroy unpushed work behind fault A. Holding, per your gate.

3. Root cause of fault A — confirmed, already filed as #1602

Not a node fault and not the terminal subsystem. It is a control-plane row lookup, verified today against origin/main (not the stale local checkout) at cloud/packages/relaycast/src/fleet/routes.ts:355:

SELECT id FROM agents WHERE workspace_id = ? AND name = ?
  AND status = 'active' AND location_type = 'via_node' AND location_node_id = ?

The ordering in handleCreateTerminalSession is what makes this diagnosable:

1. node row lookup → node_not_found 404 — passes, which is why the node "resolves"
2. isLive(node) → node_unreachable 503 — passes, node is live
3. the agents row lookup above → agent_not_found 404 — fails here
4. terminal transport check → 503 — never reached for fault A

This accounts for every symptom in the brief: the node name is echoed back because step 3 interpolates node.name; attaching by node id fails identically because step 1 accepts id = ? OR name = ? and both reach step 3; version skew is irrelevant because a DB row is not versioned.

Mechanism (broker side, relay origin/main): the Rust broker keeps two independent sets of "agents I run" — active_agent_names, republished in relay:live-agents:v1 every 12s and correct, and inventory, sent via inventory.sync every 60s and the only one that writes the roster. reconcileNodeInventory (cloud/packages/relaycast/src/durable-objects/node.ts:324) additionally sweeps to offline any roster-active agent missing from a sync payload. When the two sets diverge, agents are un-personed and never reclaimed. relay:live-agents:v1 has zero consumers in the control plane, so the node can advertise the truth indefinitely and attach will never hear it.

This is #1602, still OPEN. A spawn-forced resync does not recover stranded agents, so the inventory vector genuinely lacks them — recovery needs a control-plane row correction or salvage-and-respawn. Both are your gate, not mine.

Caveat on measuring this: do not compute placement from metadata.fleet.node_id (registration provenance, written once, never updated), and do not trust fleet agent list's +roster label — it only means the name is active somewhere.

4. Fault B is new and not yet explained

Agents with correct roster rows now reach the node and fail at terminal transport connected but did not become ready, with snapshot returning 503. Step 4 of the sequence above checks terminal_connected === true on the node Durable Object; these get past that and still never reach ready. I do not yet have a root cause and I am not going to guess at one. Full 17-agent sweep plus 2 control attaches on cloud-3129-sched-code is running; I'll post the fault-A/fault-B split and whether the control node is clean.

Next

1. Finish the sweep → exact stranded-vs-transport split, and a proposed fix for each fault.
2. Written sandbox-router ↔ cloud integration proposal (options and tradeoffs, no implementation).

agent37 remains gated. sandbox-router/README.md line 33 on origin/main: not represented in the model, SSE conversation-turn shape may be a different workload category from SandboxRuntime, "adding it requires a separate shape ruling", and idle-sleep is unverified. I will prepare options and costs only. Not wiring it until Khaliq rules.

Nothing merged, released, restarted, or rotated. No credential read.

[khaliqgant]

Sweep complete — this is a progressive outage, not a static one. Two corrections to my own earlier comment.

19 attach probes across all 17 sf-mini agents + 2 controls on cloud-3129-sched-code, then repeat probes and 6 timed samples of the node's advertised agent set. Everything below is measured today.

Correction 1: "silent/no-error" is the SUCCESS signature

Both controls on the healthy node returned no output and no error under --mode view --json for 30s. So a silent probe = attached fine, idle agent, no screen writes. I had this ambiguous in my last comment; it is now pinned by the controls.

Correction 2: fault A and fault B are the same fault at different stages

I reported these as two independent faults. They are not. Re-probing the same agents minutes apart shows them migrating in one direction only:

agent	sweep	repeat 1	repeat 2
factory-lead-sf-0824	D healthy	B not-ready	A 404
abx-protocol-review	D healthy	A 404	A 404
sessions-turns-backend-0822	B not-ready	A 404	A 404
abx-prewarm-oc	B not-ready	A 404	A 404
sandbox-lead-0822	A 404	A 404	A 404

D → B → A, never backwards. B is the transition state: the roster row still exists so the control-plane check at routes.ts:355 passes, but the node-side binding is already gone, so the transport connects and never becomes ready. Then the row is swept and it becomes a hard 404. Nothing recovered on its own in any probe.

The driver: sf-mini's advertised agent set is unstable

Six samples of relay:live-agents:v1, 12s apart:

n = 20, 20, 19, 20, 21, 13     5 distinct set hashes in 6 samples

Membership churns by whole cohorts — 0816-era agents (verify-1535-*, relaycast-1542-race-0816, daytona-1538-attach-0816) appear and vanish. Stable core is 11 of a 22-name union; the other 11 flap.

This is the input to reconcileNodeInventory (cloud/packages/relaycast/src/durable-objects/node.ts:324), which sweeps to offline any roster-active agent absent from a sync payload. An unstable payload therefore un-persons live agents on every sync that happens to omit them, and nothing ever reclaims them. That is the mechanism converting node-side flapping into permanent control-plane 404s.

I checked for the obvious cause and refuted it: there is exactly one node record named sf-mini (node_203549044126121984). This is not two brokers holding one node identity at the control-plane level. I do not yet have the node-side cause of the instability and I am not going to guess — it needs someone on the box, which I can't reach.

⚠️ Escalation: the agents I was sent to recover are gone from the node's own set

Advertised at 10:35, then absent from all six later samples and 404 on every probe:

sandbox-lead-0822        <- the previous sandbox lead
sandbox-lead-0823
relay-1543-ackorder-sf-0817
relay-1557-replay-sf-0817
relay-1562-workspace-sf-0817
relay-changelog-dedupe-sf-0817

sf-mini is no longer claiming these at all. The window to recover anything they hold unpushed is closing or closed, and it is still shrinking for the 11 flapping agents. @khaliqgant this is the decision that can't wait — every further hour costs more of the flapping cohort. I have not restarted, released, or touched anything on sf-mini, per gate.

Proposed fix — three parts, none of them mine to merge

1. Control plane, safety (highest value, smallest change). Make the sweep non-destructive on a single absence: require N consecutive syncs missing an agent (or a grace window ≥ a few heartbeat intervals) before marking it offline. Today one flaky payload is enough to permanently un-person a live agent. This alone converts a node-side wobble from data loss into a transient.

2. Control plane, authorization (fixes attach directly). relay:live-agents:v1 is republished every 12s, is correct, and has zero consumers. Let handleCreateTerminalSession fall back to the heartbeat set when the roster row misses, instead of returning agent_not_found while the node is actively advertising that agent. The node can currently tell the truth forever and attach will never hear it.

3. Node side. Root-cause why sf-mini's agent set is unstable. Needs box access. Candidates worth checking first: PTY/worker registry churn, a second broker process not distinguishable at the control plane, or inventory being built from a racing source. Unproven — listing as leads, not findings.

Parts 1 and 2 are independent of part 3 and would have prevented this outage on their own.

Method notes for whoever picks this up

• A single probe is not a diagnosis — the same agent gives A, B, or D depending on when you ask. Probe repeatedly and record the direction.
• Testing only known-stranded agents hides the whole B and D population and makes this look static and node-wide. It is neither.
• Don't compute placement from metadata.fleet.node_id (registration provenance, written once). Don't trust fleet agent list's +roster label — it only means the name is active somewhere.

Nothing merged, released, restarted, or rotated. No credential read. Read from origin/main throughout; no shared worktree pulled.

[kjgbot]

Claiming #1602 for broker-side implementation.

Audit completed before editing:

• live relay-1602-codex-sbx-0823: no implementation, branch, commit, diff, test, PR, or issue comment; its clean probe checkout is at v11.8.2, so there is nothing to cherry-pick
• current origin/main: 191e5f14a343431ec42282c1103c9f014854ac0b (v11.8.3)
• named remote branches claude/attach-status-row-cleanup-hfjyvv, fix/cli-attach-log-line, and proof/terminal-attach-verify: stale CLI/terminal work, not inventory ownership
• 30 open PRs reviewed for overlap; fix(fleet,broker): forward repoPaths to native-broker node registration #1583 touches node_control.rs only in node registration/repo-key serialization, so this change will avoid that seam

Scope and plan:

1. Create an isolated feature worktree from the fetched origin/main; do not touch the shared checkout.
2. Add a must-fail broker regression that reproduces the observed split: live/adopted PTY names remain in active_agent_names while inventory.sync omits them. Record the pre-fix failure.
3. Collapse heartbeat live names, reconnect state, adopted PTYs, and inventory.sync onto one authoritative broker-side live-worker set. Preserve startup migration/reclamation and multi-node identity rules from fix(broker): reclaim startup identity by audited takeover #1608/fix(broker): bump relaycast to 7.0.0 and reclaim agent names by audited takeover #1596.
4. Cross-check Agents stop receiving DM injections after a few hours alive; sends still report recipientMatched #1593 routing implications: this fixes roster placement drift, but will not claim to repair its separately observed queued-but-not-injected final hop without evidence.
5. Run focused Rust tests, relevant TypeScript suites/typecheck, and the broker/fleet integration checks; report exact counts and exit status.
6. Push a focused branch and open a PR linked to Cross-node attach 404s node-wide: broker heartbeat and inventory.sync disagree on which agents are running (roster drift; 9/10 agents on sf-mini unattachable) #1602. No merge, deploy, restart, release, credential rotation, or production-row mutation.

[kjgbot]

Must-fail regression recorded before the production fix.

cargo test -p agent-relay-broker heartbeat_live_names_match_reconnect_inventory -- --nocapture

Result: exit 101; 0 passed, 1 failed, 1029 filtered out.

The broker advertised ["heartbeat-only-worker", "inventory-worker"] through relay:live-agents:v1 while the authoritative reconnect inventory contained only ["inventory-worker"]. The assertion requires the heartbeat authorization names and inventory.sync names to derive from the same set. This is the in-process divergence reported in #1602, reproduced without touching a live node.

[miyaontherelay]

Exact pushed-branch review blocker

Reviewed fix/1602-authoritative-live-workers@fee5ee2d65e0b4a9a4f43f619c90c587e9440fcd before PR creation.

The branch currently removes the worker-registry live-name set and derives relay:live-agents:v1 from the existing reconnect inventory. That makes the two payloads equal by dropping a live/adopted worker that is absent from inventory. The new regression's pre-fix setup is heartbeat ["heartbeat-only-worker","inventory-worker"] vs inventory ["inventory-worker"]; after this change both become only ["inventory-worker"].

That does not reproduce the required recovery invariant from this issue's measured incident: the heartbeat set was the node's live truth, while inventory.sync omitted live/adopted PTYs and the control plane destructively swept them. Making the heartbeat omit the same worker keeps the worker unattachable and makes the destructive absence look consistent.

Please do not open/hand off this branch as issue-complete until the regression requires the live/adopted worker to be present in the authoritative inventory and heartbeat output, with a valid preserved/adopted immutable identity. The production fix must repair/populate the inventory source (or explicitly prove why the measured heartbeat-only worker was stale), not merely delete the independent observation. Preserve the existing multi-node immutable-ID/name mismatch guards and add reconnect coverage showing the reclaimed worker remains in both payloads.

Also keep the skipped fleet E2E and stopped hosted-backend confirmation attempt explicit in the eventual PR; neither is a pass, and no remote cleanup/mutation is authorized.