Skip to content

feat(llc): agent run replay — record and replay heartbeat runs for debugging and auditing #9034

Closed
Closed
feat(llc): agent run replay — record and replay heartbeat runs for debugging and auditing#9034
Assignees
mrveiss
Labels
agentsbackendenhancementNew feature or requestfeaturepriority: medium

Description

@mrveiss
mrveiss
opened on May 30, 2026

Summary

When an LLC agent produces unexpected output or fails, there is no way to replay the run to understand what happened. Replay lets developers and operators re-execute a recorded run deterministically for debugging, testing prompt changes, or audit investigations.

What Users Want

  • Each heartbeat run's inputs (context package, work item, tool calls, responses) recorded to an immutable replay log
  • UI to browse and replay any past run: re-execute with same inputs, compare output
  • Step-through mode: pause between tool calls to inspect intermediate state
  • Fast-forward: replay at 10x speed with output diff highlighted
  • Export run as a test fixture (for regression testing prompt changes)
  • Redact PII from replay logs before sharing

AutoBot Fit

autobot-backend/llc/models/heartbeat_run.py already stores run metadata. Extend with a RunReplayLog table capturing: inputs at invocation, each tool call + response, final output. New /api/llc/runs/{id}/replay endpoint that feeds stored inputs back through the adapter. Frontend: replay panel in the heartbeat run detail view.

Acceptance Criteria

  • Heartbeat run inputs serialized and stored on completion
  • Replay API re-executes run with stored inputs
  • Step-through mode pauses between tool calls
  • Output diff shown between original and replayed run
  • Export run as JSON test fixture
  • Admin-only access; PII redaction flag

Activity

  1. mrveiss
    added
    enhancementNew feature or request
    priority: medium
    feature
    backend
    agents
    on May 30, 2026
  2. mrveiss
    self-assigned this
    on Jun 4, 2026
  3. mrveiss
    added 2 commits that reference this issue on Jun 12, 2026

    fix(llc/frontend): keep replay-log query literal outside interpolatio…

    9162c59

    feat(llc): agent run replay — record, replay, diff, fixture export (#…

    bb886f0
  4. mrveiss
    closed this as completedon Jun 12, 2026

  5. mrveiss commented on Jun 12, 2026

    @mrveiss
    mrveiss
    on Jun 12, 2026
    OwnerAuthor

    Closed — merged to Dev_new_gui via PR #10030 (squash commit bb886f0d5).

    Acceptance criteria:

    • ✅ AC1 inputs serialized on completion — LLCRunReplayLog (migration 20260612_055, has_table-guarded); recording hook on the scheduler's terminal path is fire-and-forget (GC-anchored task, own session factory, asyncio.to_thread file I/O) and keyed to the adapter's exact external_run_id output path — a concurrent run can never poison another run's record
    • ✅ AC2 replay API — POST /api/llc/agents/{id}/runs/{id}/replay re-dispatches with stored inputs as a NEW linked run; gated on tenant (org-context 404), agent active status (feat(llc/budget): token-based budget mode — track agent budgets in token counts alongside dollar amounts #8997 hard-stop pause respected, 409), budget check, and a 409 while a RUNNING/QUEUED run exists; replay-mode dispatch never --resumes a live session
    • ✅ AC3 step-through — delivered as recorded-timeline step-BROWSING (documented delta: real-time pause between tool calls is physically impossible for detached CLI subprocess adapters)
    • ✅ AC4 output diff — GET .../diff/{other} (difflib unified), wired into the replay panel
    • ✅ AC5 fixture export — GET .../fixture, always credential-redacted, authenticated blob download
    • ✅ AC6 admin-only + redaction — all endpoints OWNER/ADMIN-gated via the now-canonical llc/deps.require_board_role (controls.py delegates too); ?redact_pii=true on replay-log; redact_dict list-recursion gap fixed

    179 tests; independent review (3 HIGH / 7 MEDIUM / 5 LOW — all addressed); smoke-test + code-quality + llc-e2e + llc-wiring-audit all green.

    Discoveries: #9992 was filed earlier for the stderr-PIPE recall limitation this feature inherits; pre-existing unmounted api/analytics_engagement.py surfaced by the wiring audit (predates this PR — relicense-era file).

  6. mrveiss
    added a parent issue on Sep 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

agentsbackendenhancementNew feature or requestfeaturepriority: medium

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions