Skip to content

LLC adapters: no stall or first-output watchdog — a hung run holds a slot for the full 3600s timeout #13099

Description

@mrveiss

Problem

An LLC agent run that hangs after spawning is not detected until the full
run timeout
expires — 3600 s by default. A process that is alive but producing
nothing is indistinguishable from one that is working, so a wedged agent holds a
scheduler slot and its budget allocation for an hour.

Evidence

autobot-backend/llc/adapters/subprocess_base.py:194-215 — _status() does
exactly two things:

  1. compares time.time() - started_at against one timeout_seconds
    (default ADAPTER_TIMEOUT_SECONDS = 3600, subprocess_base.py:42)
  2. returns probe_pid(pid) — a bare os.kill(pid, 0) liveness check

Nothing inspects the output file's size or mtime. There is no first-output
deadline and no stall deadline.

Proposed approach

The runs are detached and file-backed, so the output file's mtime is a free
liveness signal — no in-process watchdog is needed.

Add two deadlines alongside the existing overall timeout:

  • first-output — no bytes written within N seconds of spawn → the agent never
    started answering
  • stall — no bytes written for M seconds after output began → the agent
    stopped mid-run

Both resolved through the existing 3-tier hierarchy in resolve_timeout()
(subprocess_base.py:107-119): per-agent adapter_config override → env var →
per-adapter default. New state-file fields carry them.

The two conditions must be distinguishable in the status, because they mean
different things operationally ("never started" usually = misconfiguration or
sign-in; "stopped mid-run" usually = a hung tool call).

Risk to manage

This introduces a new false-positive class: a legitimately quiet phase — a long
build or test run with buffered stdout — would now be killed. The deadlines must
be configurable per adapter and default generously, and buffering behaviour of
each CLI should be checked before picking defaults.

Acceptance criteria

  • _status() detects first-output and stall conditions from output-file mtime
  • Both deadlines resolve through the existing 3-tier resolve_timeout hierarchy
  • The two conditions are distinguishable from each other and from the overall
    timeout in the returned status / logged reason
  • Defaults chosen against each CLI's actual output-buffering behaviour, with
    the reasoning recorded in a comment
  • Tests: never-produces-output, stalls-after-output, and a legitimately quiet
    long run that must NOT be killed

Depends on

Cancellation must reliably kill the whole process tree first, or a stall-kill
just orphans the tree faster.

Activity

  1. mrveiss commented on Jul 31, 2026

    @mrveiss
    OwnerAuthor

    Parent: #13096 (capability audit umbrella).

  2. added this to the v0.9.0 milestone on Sep 12, 2026
  3. mrveiss commented on Sep 14, 2026

    @mrveiss
    OwnerAuthor

    Closure audit against origin/main (vehicle merge #16702, closing PR #16284).

    AC Verdict Evidence
    _status() detects first-output/stall from output-file mtime met autobot-backend/llc/adapters/subprocess_support.py:check_output_stall() compares os.stat(output_file).st_size/st_mtime against first_output_deadline/stall_deadline; called from subprocess_base.py:_stall_reason() inside _status()
    Both deadlines resolve via existing 3-tier resolve_timeout hierarchy met subprocess_base.py:resolve_first_output_deadline() / resolve_stall_deadline(), same per-agent → env → per-adapter default order as resolve_timeout()
    Conditions distinguishable from each other and overall timeout met check_output_stall() returns distinct strings ("stalled: no output within ...of start" vs "stalled: no output for ...s"), separate from the existing timeout path
    Defaults reasoned against CLI buffering behaviour, recorded in comment met subprocess_base.py comment above FIRST_OUTPUT_DEADLINE_SECONDS = 120 / STALL_DEADLINE_SECONDS = 600 explains the reasoning (line-buffered JSONL/text output, generous enough to outlast one legitimate quiet stretch)
    Tests: never-produces-output, stalls-after-output, legitimately-quiet-not-killed met tests/test_stall_watchdog_13099.py::test_never_produces_output_trips_first_output_deadline, ::test_stall_after_first_output_kills_group, ::test_legitimately_quiet_within_deadline_is_not_killed

    All 5 criteria met. Kept closed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions