Repository navigation
orchestration: the fan-in wait is memory-resident at all four fan-out sites — a restart strands the batch #17281
Description
Part of #17275.
All four parent→child fan-out sites hold the join in an asyncio primitive. Nothing records that a
parent is waiting for N children, so a process restart mid-fan-out loses the join.
| Site | Wait held in |
|---|---|
autobot-backend/agents/hierarchical_agent.py:392 |
results = await asyncio.gather(*coros, return_exceptions=True) |
autobot-backend/orchestration/subagent_dispatcher.py:118 |
task_results = await bounded_gather(list(coros), self.max_parallel) |
autobot-backend/orchestration/execution_strategies/_collaborative.py:57-59 |
for task, future in task_futures: result = await future |
autobot-backend/orchestration/execution_strategies/_parallel.py:79-82 |
asyncio.wait([...], return_when=asyncio.FIRST_COMPLETED) |
Completed child results survive — orchestration/checkpoint_resumer.py:39-50 persists each
finished step and workflow_executor.py:719-720 skips already-completed steps on an explicit
resume. What does not survive is the coordination state, so recovery needs an out-of-band call
rather than happening automatically at startup.
Why this blocks enablement
With AUTOBOT_DELEGATION_ENABLED off (chat_workflow/delegation.py:69) no production traffic
reaches these paths, so the defect is latent. Turning delegation on makes it live.
Shape of the fix
Record the join before awaiting, settle it per returning child, and re-derive open joins on boot
— the pattern already shipped at initialization/lifespan.py:2043 / :2058.
- Persist
(parent, [child ids], status)in the same transaction that dispatches the children - Settle one child per terminal return with a guarded conditional update, not a blind write
- Add a bounded, idempotent startup scan for joins left open, alongside the two existing ones
Acceptance criteria
- A fan-out's pending join is readable from a durable store while children run
- Killing the process mid-fan-out and restarting re-derives the join with no out-of-band call
- Child settlement is idempotent — a duplicate return cannot double-settle
- The startup scan is bounded and cannot loop on a non-progressing page
Activity