Repository navigation
Umbrella: delegation's fan-in wait does not survive a restart — durability before enablement #17275
Description
Agent delegation is built, governed, tested — and switched off. The reason it cannot simply be
switched on is that the fan-in wait (a parent waiting for its N children) is memory-resident at
every site. A restart mid-fan-out strands the batch, and nothing re-derives it.
This umbrella groups the durability work that is a precondition for setting
AUTOBOT_DELEGATION_ENABLED=true, not an independent improvement.
Found during a comparative architecture review against an external orchestration runtime; full
analysis and evidence in docs/research/durable-subagent-orchestration-controller-state.md.
The shape of the defect
The four fan-out sites all hold the wait in an asyncio primitive:
| Site | Wait |
|---|---|
autobot-backend/agents/hierarchical_agent.py:392 |
asyncio.gather |
autobot-backend/orchestration/subagent_dispatcher.py:118 |
bounded_gather |
autobot-backend/orchestration/execution_strategies/_collaborative.py:57-59 |
resident await future loop |
autobot-backend/orchestration/execution_strategies/_parallel.py:79-82 |
asyncio.wait |
Completed child results are checkpointed (orchestration/checkpoint_resumer.py:39-50), so
finished work survives. The coordination state — "this parent is waiting for these N children" —
is not written anywhere, so a crash loses the join and requires an out-of-band resume.
We already own the corrective pattern in two places: initialization/lifespan.py:2043
(_recover_agent_sessions → llc/scheduler/session_checkpointer.py:112 recover_incomplete_runs)
and lifespan.py:2058 (_recover_index_queue). The work is to extend it, not invent it.
Tasks
- Task 1: persist the fan-in join so a restart can re-derive it — orchestration: the fan-in wait is memory-resident at all four fan-out sites — a restart strands the batch #17281
- Task 2:
SubagentDispatcheris reachable only behind a default-off flag — orchestration: SubagentDispatcher is reachable only behind a default-off flag — decide enable, document, or retire #17282 - Task 3:
ProcessAdapterServicehas no startup recovery — reliability: ProcessAdapterService has no startup recovery — a crashed run leaves its row at RUNNING forever #17283 - Task 4:
LongRunningOperationManagercannot resume after a restart — reliability: LongRunningOperationManager.resume_operation always fails after a restart — the checkpoint survives, the registry does not #17284 - Task 5: governed delegation was never actually enabled — delegation: governed delegation was never enabled — #11266 closed on a hardening PR with every checklist box unticked #17285 (blocked by orchestration: the fan-in wait is memory-resident at all four fan-out sites — a restart strands the batch #17281)
- Task 6: the superseded delegation pair is still present — delegation: the superseded HierarchicalAgent/DelegateTool pair is still present with no production caller #17286
- Task 7: bound the whole delegation tree (a cap that survives into children, a timeout for the internal engine, a scripted chain test) — delegation: the per-turn cap resets in every child and the internal engine never times out — bound the whole tree before enablement #18134 (blocks delegation: governed delegation was never enabled — #11266 closed on a hardening PR with every checklist box unticked #17285)
Tasks 3 and 4 extend the class named in #14863 from three in-memory job stores to five — the
two below were never counted. #1751 (closed) already proposed consolidating ProcessAdapterService
with the long-running-operations framework.
Adjacent observation (not a task)
llc/scheduler/session_checkpointer.py:138-140 performs an unconditional UPDATE on recovery.
It is correct today only because the startup scan is single-threaded — there is no
WHERE status='running' guard at the DB level. Filed separately as a latent risk — see the standalone list below.
Related
- Three in-memory job stores solve process-death to three different degrees — the wizard's has no bound at all #14863 — three in-memory job stores; this umbrella adds two more to the same class
- Consolidate ProcessAdapterService (#1406) with long_running_operations framework #1751 (closed) — consolidate ProcessAdapterService with long_running_operations
- AgentLoop (agent_loop/) is never instantiated in production — decide wire vs port vs document #11221 (closed) — AgentLoop has no production caller; the live seam is
chat_workflow - POST /tasks/{task_id}/steer has a live route and no consumer -- wire or retire #16993 — a live route whose guidance nothing reads, same root
- [Umbrella] AutoBot Efficiency & Precision — wire built-but-disconnected machinery + runtime fixes #10603 — umbrella for built-but-disconnected machinery
Activity
docs(research): three review corrections on the loop audit (#17271)
docs(research): land the durable subagent orchestration analysis (#17275
docs(research): scrub the source's own stack and paths per the 2026-0…
docs(research): fold the two comparative research docs into this PR (#…
fix(pricing): migrate LLM cost tracking off hardcoded price tables (#…
docs(research): land the scrub that #17296 closed without merging (#1…
docs(research): carry a handed-over source analysis, scrubbed (#17275)
docs(research): make both reports honour the omissions they claim (#1…
docs(research): the header said the comparison had not started; it is…
fix(security): scope agent, terminal and chain-of-thought events to t…