Skip to content

Umbrella: delegation's fan-in wait does not survive a restart — durability before enablement #17275

Open
0 of 8 issues completed

Description

@mrveiss

Agent delegation is built, governed, tested — and switched off. The reason it cannot simply be
switched on is that the fan-in wait (a parent waiting for its N children) is memory-resident at
every site. A restart mid-fan-out strands the batch, and nothing re-derives it.

This umbrella groups the durability work that is a precondition for setting
AUTOBOT_DELEGATION_ENABLED=true, not an independent improvement.

Found during a comparative architecture review against an external orchestration runtime; full
analysis and evidence in docs/research/durable-subagent-orchestration-controller-state.md.

The shape of the defect

The four fan-out sites all hold the wait in an asyncio primitive:

Site Wait
autobot-backend/agents/hierarchical_agent.py:392 asyncio.gather
autobot-backend/orchestration/subagent_dispatcher.py:118 bounded_gather
autobot-backend/orchestration/execution_strategies/_collaborative.py:57-59 resident await future loop
autobot-backend/orchestration/execution_strategies/_parallel.py:79-82 asyncio.wait

Completed child results are checkpointed (orchestration/checkpoint_resumer.py:39-50), so
finished work survives. The coordination state — "this parent is waiting for these N children" —
is not written anywhere, so a crash loses the join and requires an out-of-band resume.

We already own the corrective pattern in two places: initialization/lifespan.py:2043
(_recover_agent_sessions → llc/scheduler/session_checkpointer.py:112 recover_incomplete_runs)
and lifespan.py:2058 (_recover_index_queue). The work is to extend it, not invent it.

Tasks

Tasks 3 and 4 extend the class named in #14863 from three in-memory job stores to five — the
two below were never counted. #1751 (closed) already proposed consolidating ProcessAdapterService
with the long-running-operations framework.

Adjacent observation (not a task)

llc/scheduler/session_checkpointer.py:138-140 performs an unconditional UPDATE on recovery.
It is correct today only because the startup scan is single-threaded — there is no
WHERE status='running' guard at the DB level. Filed separately as a latent risk — see the standalone list below.

Related

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions