Skip to content

fix(agent): pre-action verifier and belief state run nowhere — AgentLoop has no production caller #14031

Description

@mrveiss

Child of #13587.

Problem

The pre-action adversarial verifier and the belief-state tracker both run in
production nowhere. Their only caller is AgentLoop, which has no production
instantiation — the live path is the LangGraph StateGraph in
chat_workflow/graph.py plus chat_workflow/tool_handler.py.

Evidence

grep -rn "from agent_loop.loop import\|AgentLoop(" across the repo (excluding
worktrees and venv) returns only test files:

  • autobot-backend/agent_loop/test_loop_repetition.py:22,44,212
  • autobot-backend/agent_loop/test_approval_workflow.py:25,53
  • autobot-backend/agent_loop/spill_loop_wiring_test.py:183,185
  • autobot_shared/tool_catalogue_test.py:76,194

Both components are reachable only from that dead entry point:

  • PreActionVerifier — sole non-test caller is agent_loop/loop.py:191-192,1668.
    pre_action_verifier_enabled defaults to True (agent_loop/types.py:266),
    so it reads as an active guard while executing on no request.
  • BeliefState / BeliefStateUpdater — sole non-test caller is
    agent_loop/loop.py:28,186,644,699,1350,1385. belief_state_enabled defaults
    to False (agent_loop/types.py:258): off and unreachable.

Correction to this umbrella's own premise

#13587's "Already covered — audited, confirmed present, NOT filed" section lists:

Pre-action adversarial verifier — agent_loop/pre_action_verifier.py,
verdict surfaced to the human at the approval gate (agent_loop/loop.py:1600,1676).
No external equivalent.

The citation is to a module with no production caller. The capability is real and
well built; it is not covered, because it never executes. The same correction
applies to the "Durable run checkpointing" entry, which also cites agent_loop/loop.py.

By contrast, the entries citing chat_workflow/tool_handler.py are accurate and
verified still wired today: fact-forcing at tool_handler.py:3096 and config
protection at tool_handler.py:3061.

The pattern that already worked

#13590 (CLOSED) is the template. The repetition/stagnation guard had the identical
problem, and the fix was not to wire AgentLoop in — it was to extract the
decision logic into autobot_shared/repetition_guard.py as a dependency-free pure
function and call it from the live seam (tool_handler.py:3163). Its own module
docstring records the diagnosis: the machinery "existed in agent_loop/ and ran
nowhere". fact_forcing took the same route and is now a thin re-export of
autobot_shared.fact_forcing_guard.

Keep the decision logic pure and side-effect-free; let the graph node own the side
effect. That is what makes it both testable and wireable.

Scope

  • Extract PreActionVerifier's decision surface into autobot_shared/,
    preserving the HARD_BLOCK verdict, and call it from the live approval gate.
  • Extract the belief-state update path the same way; wire it into the tool seam
    where tool results are already collected.
  • Decide belief_state_enabled's default deliberately once it is reachable —
    flipping it while unreachable changes nothing and would read as a fix.
  • Add a wiring assertion per guard: a test that fails if the live seam stops
    calling it. Without this the next refactor silently un-wires it again, and
    "no failures" reads as "working".

Related, same root cause, already filed: #13919 (read_spilled_output has no
production callers) and #13997 (tool-output spill wired only into AgentLoop).
Those two plus this one are the full inventory of agent_loop/ capability that
never reaches production.

Order

Wave B of #13587, after #13590 (CLOSED — it establishes the extraction
pattern). No dependency on #14027/#14028/#14029. Coordinate with #13919/#13997:
one extraction pass over agent_loop/ is cheaper than three.

Found during a comparative audit of an external agent-harness architecture, where
guards are pure decision classes plugged into graph nodes so there is only one
runtime and no second harness to fall out of date.

Activity

  1. mrveiss commented on Aug 11, 2026

    @mrveiss
    OwnerAuthor

    Full audit context: docs/research/agent-harness-guard-and-context-audit.md — landing in PR #14032.

    The doc records what AutoBot already covers (with file:line evidence, so this issue is not re-audited later), where our design is deliberately stronger than the source's, and the do-not-re-file list — alongside all five gaps (#14027, #14028, #14029, #14030, #14031) and their wave order.

  2. mrveiss commented on Aug 11, 2026

    @mrveiss
    OwnerAuthor

    Correction — this is not a new discovery; #11221 already documented it

    A code-reviewer pass caught a material omission in the body above.

    The dormant loop is known, in-code, under a CLOSED issue

    autobot-backend/agent_loop/__init__.py:11-15 already says:

    NOT WIRED IN PRODUCTION (#11221): AgentLoop is never instantiated by any production
    caller — the live tool seam is chat_workflow._dispatch_tool_call. Submodules here
    (belief_state, extractors, guards) ARE reused by other subsystems, but the loop
    orchestrator itself is dormant, so anything wired only into it is inert.

    #11221 is CLOSED — "AgentLoop (agent_loop/) is never instantiated in production — decide
    wire vs port vs document", resolved as document. The body above presents the dormancy as
    freshly discovered while correcting #13587, and never cites #11221. That is exactly the kind
    of omission that makes an issue look bigger than it is.

    What is actually new — and it is narrower

    1. pre_action_verifier_enabled defaults to True (agent_loop/types.py:266) inside that
      dormant module. A disabled-by-default dormant component is honest; an enabled-by-default one
      reads as an active guard while executing on no request — and is very likely why Umbrella: agent-seam governance is name-scoped, and the best guards are not on the production path #13587
      miscounted it. belief_state_enabled defaults to False (:258), so it is off and
      unreachable, which is the less misleading of the two.
    2. Umbrella: agent-seam governance is name-scoped, and the best guards are not on the production path #13587's "Already covered — NOT filed" list miscounts both, citing agent_loop/loop.py
      for the pre-action verifier and for durable run checkpointing. Documenting a dormant module
      under AgentLoop (agent_loop/) is never instantiated in production — decide wire vs port vs document #11221 did not prevent a later audit from reading it as live coverage — which is the
      real lesson, and the argument for extraction over documentation.

    Grep wording corrected

    grep -rn "from agent_loop.loop import\|AgentLoop(" returns two non-test hits besides the
    tests: agent_loop/__init__.py:38 (package re-export) and :28 (docstring usage example).
    Neither is an instantiation, so the conclusion holds — but "returns only test files" was wrong
    as stated.

    What this changes about the work

    Nothing about the scope; something about the framing. #11221 chose document, and the
    documentation demonstrably failed to stop a downstream audit from treating the module as live.
    That is the case for choosing extract-and-wire this time rather than documenting again —
    and for the wiring assertion in the acceptance criteria, which is the only form of
    documentation that fails loudly.

    Full corrected evidence: docs/research/agent-harness-guard-and-context-audit.md, Gap 5
    (PR #14032).

  3. mrveiss commented on Aug 11, 2026

    @mrveiss
    OwnerAuthor

    First slice landed: PR #14119 extracts and wires only PreActionVerifier — belief-state extraction and the belief_state_enabled default decision are deliberately left for a follow-up (belief-state's flag defaults False, so it's off and unreachable, a smaller/separable problem than the verifier's default-True-but-inert one).

    Summary of #14119:

    • autobot_shared/pre_action_verifier_guard.py — the extracted decision surface (pure: threshold resolution, response parsing, determine_verdict/panel_decision — the HARD_BLOCK-preserving part) plus the PreActionVerifier class (the LLM call itself can't be made pure).
    • agent_loop/pre_action_verifier.py is now a thin re-export (the fact_forcing precedent).
    • chat_workflow/tool_handler.py gets a new _enforce_pre_action_verifier enforcer in the live dispatch chain, scoped to SENSITIVE_TOOLS, gated on pre_action_verifier_enabled (guard-profile resolved, defaults True).
    • Wiring assertion + mutation test included, per the feat(agent-loop): production seam has no repetition/stagnation guard — the guards exist in a dormant module, unported #13590 template.

    Flagged in the PR body as a finding rather than a quiet scope change: execute_command is in SENSITIVE_TOOLS and is almost certainly the highest-volume sensitive tool in production, so this adds an LLM round trip in front of its existing approval flow for every shell command. That's the original AgentLoop scope exactly (never exercised before because AgentLoop never ran in production) — worth a latency/cost look post-merge, not something I narrowed unilaterally.

    Leaving this issue open for the belief-state slice.

  4. added
    needs-decisionBlocked on an owner decision; options and a recommendation are on the issue
    on Aug 12, 2026
  5. 3 remaining items

  6. mrveiss commented on Sep 1, 2026

    @mrveiss
    OwnerAuthor

    Half of this has been fixed since filing — one of the two components named is now live.

    PreActionVerifier — no longer dead. It has a real non-test caller:

    autobot-backend/chat_workflow/tool_dispatch_guards.py:193
        await PreActionVerifier().verify(...)
    

    imported by chat_workflow/tool_handler.py:51, plus the extracted autobot_shared/pre_action_verifier_guard.py:387. This landed via the #13590 extraction template.

    BeliefStateUpdater and AgentLoop — still dead. Non-test references to BeliefState are agent_loop/loop.py:28,186 (its own definition and use) and benchmarks/benchmark_belief_state.py:78,384, which is a benchmark script, not production. For AgentLoop:

    $ git grep -n -E "from agent_loop.loop import|AgentLoop\(" origin/Dev_new_gui | grep -v test
    0 non-test callers
    

    The single hit in agent_loop/__init__.py:28 is inside the module docstring's usage example, not code.

    So the remaining scope is BeliefStateUpdater + AgentLoop, and the title's "pre-action verifier and belief state run nowhere" is now half true. Per the never-delete rule these are unfinished work to wire in, not code to remove — and PreActionVerifier is the worked example of how.

  7. added this to the v0.9.0 milestone on Sep 12, 2026
  8. mrveiss commented on Sep 17, 2026

    @mrveiss
    OwnerAuthor

    Verified against merged main. One of four delivered; the other three all hang off the same missing piece.

    AC1 — delivered, ticked

    28e9df7c8 (PR #14119) extracted the verifier and wired it at the live seam:

    autobot_shared/pre_action_verifier_guard.py 564 lines — the decision surface, out of agent_loop/
    autobot-backend/chat_workflow/tool_handler.py +99 lines — the seam now calls it
    autobot-backend/agent_loop/pre_action_verifier.py −465 lines — the dead copy shrank to a shim

    So the verifier no longer "runs nowhere": the tool path invokes it.

    AC4 — half delivered

    autobot-backend/tests/unit/chat_workflow/test_tool_handler_pre_action_verifier.py carries test_the_call_site_still_names_the_enforcer, which is exactly the wiring assertion this criterion asks for — it fails if the seam stops calling the guard.

    But the criterion says per guard, and there is only one guard wired. The belief-state half has no seam to assert against, so AC4 cannot complete before AC2.

    AC2 and AC3 — not delivered

    belief_state appears in no autobot_shared/ or chat_workflow/ file on main. Every reference is still inside autobot-backend/agent_loop/:

    agent_loop/__init__.py   agent_loop/loop.py   agent_loop/types.py
    agent_loop/text_utils.py agent_loop/tests/test_belief_cache.py
    benchmarks/benchmark_belief_state.py
    

    AgentLoop still has no production caller, so belief state is in exactly the position this issue was filed about — the verifier was lifted out of it and the belief-state path was not.

    AC3 (decide belief_state_enabled's default deliberately once it is reachable) is explicitly conditional on that reachability, so it cannot be judged, let alone met, until AC2 lands.

    One thing worth a look

    28e9df7c8 also added a tracked entry named venv (venv | 1 + in its diffstat). That looks like an accidental commit of a virtualenv symlink rather than intended content. Not this issue's subject and I have not touched it — flagging it because a tracked venv entry tends to cause confusing failures later.

    Remaining work is one extraction (AC2), which then unblocks AC3 and completes AC4. Not closing.

  9. mrveiss commented on Sep 17, 2026

    @mrveiss
    OwnerAuthor

    Correction to my previous comment's last section: the venv entry is already fixed and needs no look.

    I read it out of 28e9df7c8's diffstat and flagged it without checking the present state. It is not on main: d74d66875 — "fix(repo): untrack the venv symlink that destroys the interpreter on checkout (#14133) (#14138)" — removed it, and it had its own issue.

    So that paragraph described a commit correctly and the current tree wrongly. The two are different claims and I made the second while only checking the first. Everything above it stands: AC1 delivered and ticked, AC4 half delivered, AC2 and AC3 blocked on the belief-state extraction.

  10. mrveiss commented on Sep 20, 2026

    @mrveiss
    OwnerAuthor

    Re-verified against current main — the defect holds, and re-checking it has a trap in it

    Triaged under the branch freeze.

    The claim still holds

    AgentLoop has no production instantiation on main. PreActionVerifier and
    BeliefStateUpdater remain reachable only through agent_loop/loop.py, which nothing constructs:

    • PreActionVerifier — loop.py:28 (import), loop.py:191-192 (construction)
    • BeliefStateUpdater — loop.py:27, loop.py:186; the only other non-test reference is
      benchmarks/benchmark_belief_state.py:78,384, a benchmark, not a caller
    • Defaults unchanged: types.py:266 pre_action_verifier_enabled: bool = True,
      types.py:258 belief_state_enabled: bool = False

    So the verifier still reads as an active guard — enabled by default — while executing on no
    request at all. That is the sharpest part of this issue and it is still true.

    The trap, worth recording before someone re-verifies

    The issue's evidence section says the grep returns "only test files". Run today, it also returns:

    autobot-backend/agent_loop/__init__.py:28:    loop = AgentLoop(
    

    That is inside the module docstring — the package's own usage example, not code. A reader
    checking whether this issue is stale sees a non-test hit in a production file and can reasonably
    conclude a caller now exists and the issue is fixed. It is not; the only other reference at
    __init__.py:38 is the re-export.

    This is the same failure this issue is about, one level up: something that reads as reachable while
    reaching nothing. Anyone re-verifying should exclude docstrings, or check what the line is inside
    before counting it.

    Branch verdict: do not revive

    issue-14031 is 4,286 files divergent from main, 6 weeks old, with two unlanded paths:

    1. autobot-backend/chat_workflow/tool_handler.py — its copy predates security(agents): an agent refused an action can get another agent to do it — no per-peer identity #16950 and feat(agents): deliver peer messages at the recipient's next turn, not mid-task #16948, so taking
      it would drop both (guards(git): reviving a stale branch can silently reverse a later fix — 75 of 87 revivable branches carry one #17178).
    2. venv — a committed symlink (mode 120000) pointing at an absolute path inside a
      developer's home directory. It is absent from main, correctly, and must stay that way.

    The second one is worth naming: it is invisible in a file-count summary, survives any "only two
    files changed" reading, and would plant an absolute machine-specific path in the repository. It is
    also a second reason this branch is not a starting point for anything.

    What the fix still needs

    Per the never-delete rule, both components get wired into the live path —
    chat_workflow/graph.py and chat_workflow/tool_handler.py — rather than removed. Until then the
    honest interim state would be for pre_action_verifier_enabled to stop defaulting to True, since
    a default that advertises an inactive guard is the misleading half of the defect.

    Issue stays open. Nothing here is recoverable from the branch.

  11. mrveiss commented on Sep 25, 2026

    @mrveiss
    OwnerAuthor

    Correction: the verifier half of this issue is resolved

    Re-checked on main during a comparative research audit
    (docs/research/confidence-gated-decision-loop-harness.md).

    PreActionVerifier now has a production caller.
    chat_workflow/tool_dispatch_guards.py:154 enforce_pre_action_verifier, whose own docstring
    records it as "its first production caller", wired into the dispatch chain at
    chat_workflow/tool_handler.py:3264-3335. Scope matches the original — only tools matching the
    canonical SENSITIVE_TOOLS set — and on a soft block it threads verifier_rationale,
    verifier_refutation_probability and verifier_degradation into the pending_approval payload,
    so the rationale reaches the human.

    Still true: AgentLoop itself has no production caller. grep -rn "AgentLoop(" outside
    tests returns one hit, the docstring usage example at agent_loop/__init__.py:28.

    Not re-checked: the belief-state half of this issue.

    Adjacent defect found in the same config object and filed separately as #17469 — the three
    timeout budgets in agent_loop/types.py:189-192 are read by nothing, so wiring this loop would
    not bring its budgets with it.

  12. mrveiss commented on Sep 28, 2026

    @mrveiss
    OwnerAuthor

    v0.9.1 triage (classification only — not a fix proposal)


    Generated by Claude Code

  13. mrveiss commented on Sep 28, 2026

    @mrveiss
    OwnerAuthor

    Pre-filter + per-AC pass — partial, and the gap is extraction rather than wiring. Not closed.

    Read-only at origin/main ab05ba3aef. One commit names this issue: 28e9df7c84 "fix(agent): extract and wire the pre-action verifier at the live tool seam (#14031)" — which is AC1, already ticked.

    The belief-state half (AC2) is wired but not extracted. belief_state_enabled is consulted in the live loop at agent_loop/loop.py:644, :699 and :1350, so the path is reachable — but the criterion asks to "extract the belief-state update path the same way", i.e. into autobot_shared/, as the verifier was. A search for belief_state/BeliefState under autobot_shared/ finds nothing. So the asymmetry this issue was filed about is half closed: the verifier is shared and wired, the belief state is wired in place.

    That distinction matters for AC4 (a wiring assertion per guard: a test that fails if the live seam stops). A guard over a shared module can assert the seam from outside; a guard over a path that only exists inside loop.py has to assert against the loop, which is the shape that rots. Extraction is the precondition for the guard, not a tidiness step — which is presumably why the criterion says the same way.

    AC3 (decide belief_state_enabled's default deliberately once it is reachable) is now answerable, since the three call sites above make it reachable. It is a decision, not work: the default is currently whatever the config says, and the criterion asks for that to be a recorded choice rather than an inherited one.

    Verdict: 1 of 4, with AC2 as the blocker for AC4 and AC3 available to decide now. Nothing here needs a host.

  14. mrveiss commented on Sep 29, 2026

    @mrveiss
    OwnerAuthor

    B7 spec — #17725 delivers most of this issue's open criteria. The gap is narrower than the batch plan assumed.

    Full spec at ~/b7-spec-2026-09-29.md. Read from origin/issue-16818-lease-wiring (PR #17725, open) against origin/main.

    I expected to specify the release path. The branch already contains it — its +28 to stalled_run_sweep.py is:

    +sweep now releases what the run held: a stalled run hands its workspace back on the
    +same pass that closes it out.
    +from llc.services.workspace_lease import release_for_run
    +STALL_RELEASE_REASON = "run stalled: workspace released by the stalled-run sweep (#16817/#16818)"
    +    released += len(await release_for_run(session, run.id, STALL_RELEASE_REASON))
    
    AC After #17725 merges
    AC2 releases what the run held lease: delivered
    AC5 two sweepers must not double-release delivered, with the reasoning in the diff — "Inside the same transaction as the status write, and under the same SKIP LOCKED selection: a run that this worker did not claim is not this worker's to release"
    AC6 release half of the test stalled_run_sweep_test.py +56
    AC7 reports what it released released counter + logger.info

    The gap: AC2 names three things and the branch releases one

    "claims, assignments, workspace leases" — no claim or assign appears anywhere in those additions.

    Change: in the same transaction and under the same SKIP LOCKED selection, release the run's claims and assignments alongside its lease, each with a reason distinguishable from an expiry the way STALL_RELEASE_REASON already is.
    Assertion: a swept run afterwards holds no claim, no assignment and no live lease, and the log names all three counts.
    Mutation: release the lease only → the claims/assignments assertion reddens while the lease assertion stays green. Without that divergence, "releases what the run held" is satisfied by releasing one of three things.

    One boundary #17725 records, which this must not cross: "The release frees the slot, not the directory. Whether the directory may be removed is a separate question with its own evidence, and it is asked in llc.services.workspace_disposal." Directory disposal is out of scope here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    agentsarea: agent-seamWave 3 · cluster H — Agent seam & tool governancebackendbugSomething isn't workingnot-wiredImplemented but not connected to apppriority: high

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions