Repository navigation
LLC adapters: no stall or first-output watchdog — a hung run holds a slot for the full 3600s timeout #13099
Copy link
Copy link
Closed
Labels
agentsai-mlarea: voice-visionWave 5 · cluster N — Voice & visionWave 5 · cluster N — Voice & visionbackendenhancementNew feature or requestNew feature or requeststability
Milestone
Description
Activity
- addedenhancementNew feature or requestNew feature or request
on Jul 31, 2026 Parent: #13096 (capability audit umbrella).
- addedarea: voice-visionWave 5 · cluster N — Voice & visionWave 5 · cluster N — Voice & vision
on Sep 1, 2026 - added 4 commits that reference this issue
on Sep 11, 2026 Closure audit against
origin/main(vehicle merge #16702, closing PR #16284).AC Verdict Evidence _status()detects first-output/stall from output-file mtimemet autobot-backend/llc/adapters/subprocess_support.py:check_output_stall()comparesos.stat(output_file).st_size/st_mtimeagainstfirst_output_deadline/stall_deadline; called fromsubprocess_base.py:_stall_reason()inside_status()Both deadlines resolve via existing 3-tier resolve_timeouthierarchymet subprocess_base.py:resolve_first_output_deadline()/resolve_stall_deadline(), same per-agent → env → per-adapter default order asresolve_timeout()Conditions distinguishable from each other and overall timeout met check_output_stall()returns distinct strings ("stalled: no output within ...of start"vs"stalled: no output for ...s"), separate from the existing timeout pathDefaults reasoned against CLI buffering behaviour, recorded in comment met subprocess_base.pycomment aboveFIRST_OUTPUT_DEADLINE_SECONDS = 120/STALL_DEADLINE_SECONDS = 600explains the reasoning (line-buffered JSONL/text output, generous enough to outlast one legitimate quiet stretch)Tests: never-produces-output, stalls-after-output, legitimately-quiet-not-killed met tests/test_stall_watchdog_13099.py::test_never_produces_output_trips_first_output_deadline,::test_stall_after_first_output_kills_group,::test_legitimately_quiet_within_deadline_is_not_killedAll 5 criteria met. Kept closed.
Metadata
Metadata
Assignees
Labels
agentsai-mlarea: voice-visionWave 5 · cluster N — Voice & visionWave 5 · cluster N — Voice & visionbackendenhancementNew feature or requestNew feature or requeststability
Problem
An LLC agent run that hangs after spawning is not detected until the full
run timeout expires — 3600 s by default. A process that is alive but producing
nothing is indistinguishable from one that is working, so a wedged agent holds a
scheduler slot and its budget allocation for an hour.
Evidence
autobot-backend/llc/adapters/subprocess_base.py:194-215—_status()doesexactly two things:
time.time() - started_atagainst onetimeout_seconds(default
ADAPTER_TIMEOUT_SECONDS = 3600,subprocess_base.py:42)probe_pid(pid)— a bareos.kill(pid, 0)liveness checkNothing inspects the output file's size or mtime. There is no first-output
deadline and no stall deadline.
Proposed approach
The runs are detached and file-backed, so the output file's mtime is a free
liveness signal — no in-process watchdog is needed.
Add two deadlines alongside the existing overall timeout:
started answering
stopped mid-run
Both resolved through the existing 3-tier hierarchy in
resolve_timeout()(
subprocess_base.py:107-119): per-agentadapter_configoverride → env var →per-adapter default. New state-file fields carry them.
The two conditions must be distinguishable in the status, because they mean
different things operationally ("never started" usually = misconfiguration or
sign-in; "stopped mid-run" usually = a hung tool call).
Risk to manage
This introduces a new false-positive class: a legitimately quiet phase — a long
build or test run with buffered stdout — would now be killed. The deadlines must
be configurable per adapter and default generously, and buffering behaviour of
each CLI should be checked before picking defaults.
Acceptance criteria
_status()detects first-output and stall conditions from output-file mtimeresolve_timeouthierarchytimeout in the returned status / logged reason
the reasoning recorded in a comment
long run that must NOT be killed
Depends on
Cancellation must reliably kill the whole process tree first, or a stall-kill
just orphans the tree faster.