Skip to content

fix(websockets): /api/ws broadcast is a single global callback — multi-client delivery and chat-history persistence both break #14814

Closed

Description

@mrveiss

Summary

The main event WebSocket (/api/ws) cannot serve more than one client. The broadcast handler is
held as a single scalar callback, so each new connection overwrites the previous one, and the
first disconnect clears it for everyone. A third-order effect is that chat history stops being
persisted
.

Found during a research audit comparing AutoBot's event/state layer against the Agent Host Protocol
(see Sources). This is a pre-existing defect, independent of any protocol adoption.

Root cause

autobot-backend/event_manager.py:31 declares the callback as a scalar, not a collection:

self._websocket_broadcast_callback: Callable[[Dict[str, Any]], Awaitable[None]] | None = None

register_websocket_broadcast (event_manager.py:53) assigns straight over it, and
autobot-backend/api/websockets.py:681 calls that once per accepted connection via
get_event_bus().register_ws_broadcast(broadcast_event).

Impact

  1. Last-writer-wins. Client B connecting silently stops Client A receiving anything on this
    path. No error, no warning, no log.
  2. First disconnect mutes every remaining client. api/websockets.py:694 clears the callback to
    None in the endpoint's finally — including for a connection that was already superseded.
    Clients that are still connected go permanently silent until they reconnect.
  3. Chat history stops persisting. _add_to_chat_history is invoked from inside
    broadcast_event (api/websockets.py:510, helper at :459). No registered callback means no
    broadcast, which means no history write. This is a data-loss path, not only a UI path.

Blast radius

PersistStrategy.NONE routes to this single-callback path (events/bus.py:85). Current callers:

  • api/workflow.py — 7 call sites (:562, :658, :693, :716, :843, :870, :986)
  • chat_workflow/cot_events.py:201 — chain-of-thought streaming
  • api/agent.py:194 — agent events
  • orchestrator.py:968, orchestration/workflow_runner.py:695
  • services/approval_gate_service.py:362 — approval gates
  • services/npu_worker_manager.py:902, services/load_balancer.py:649
  • worker_node.py:298,399,415, diagnostics.py:173,204,265

The frontend's primary socket connects to exactly this endpoint:
autobot-frontend/src/services/GlobalWebSocketService.ts:146,153.

Reproduction

  1. Open the GUI in two browser tabs (both establish /api/ws).
  2. Trigger a workflow or a chat turn.
  3. Only the most recently connected tab receives events.
  4. Close that tab. The remaining tab now receives nothing, and new events are no longer written to
    chat history.

Proposed fix

  • autobot-backend/event_manager.py — hold a set[Callable] instead of a scalar; turn
    register_websocket_broadcast into add/discard semantics; fan out to all registered callbacks in
    publish, removing any that raise a connection error.
  • autobot-backend/events/bus.py — widen register_ws_broadcast to match (register/unregister
    rather than set/clear).
  • autobot-backend/api/websockets.py:671-697 — register and unregister this connection's
    handler rather than overwriting a global.
  • autobot-backend/api/websockets.py:459,510 — move _add_to_chat_history out of the broadcast
    handler so persistence no longer depends on a client being attached.

Acceptance criteria

  • Two concurrent /api/ws clients both receive every published event.
  • Disconnecting one client does not affect delivery to the others.
  • Events are written to chat history when zero clients are connected.
  • Regression test asserting all three, driving the production endpoint (not the helper) and
    asserting at the far boundary — a mutation to the fan-out must fail it.

Sources

Audit context and the full comparison: docs/research/agent-host-and-agent-client-protocols.md
(PR pending on branch research-ahp-acp).

Activity

  1. mrveiss commented on Aug 22, 2026

    @mrveiss
    OwnerAuthor

    Audit trail and full analysis: #14826 — docs/research/agent-host-and-agent-client-protocols.md.

    Sources for this finding:

  2. github-actions commented on Aug 23, 2026

    @github-actions
    Contributor

    PR #14826 (merged to Dev_new_gui) references this issue with a close keyword.

    feat(events): multi-client session synchronization, replay, and an ACP agent surface

    If this issue is fully resolved, close it manually. If work remains, no action is needed.

  3. mrveiss commented on Aug 23, 2026

    @mrveiss
    OwnerAuthor

    Delivered and merged.

    event_manager.py: the broadcast handler is a set, not a scalar — several clients receive events, a failing callback is isolated and unregistered, and one disconnect no longer silences the rest. _add_to_chat_history moved OUT of the broadcast callback onto a publish-time hook wired in initialization/lifespan.py, so history is written exactly once per publish and with zero clients attached.
    Tests: event_manager_multiclient_14814_test.py (7), api/websockets_chat_history_routing_test.py (5, repointed to the new seam), api/websockets_scoping_review_test.py (10).

    Evidence

    • PR feat(events): multi-client session synchronization, replay, and an ACP agent surface #14826, squash-merged as ded8d07e7 into Dev_new_gui
    • CI on the merged head 889e3ac4a: 57 passed / 0 failed / 0 cancelled / 0 pending
    • Codecov: "All modified and coverable lines are covered by tests" — 100% patch coverage
    • Independent code-reviewer pass run before merge; all 3 findings fixed in-PR with regression tests
    • Content verified present in origin/Dev_new_gui after merge

    Not verified on a host. The tests above exercise the seams, but no running instance has confirmed the end-to-end behaviour (a second tab receiving events, a reconnect replaying the gap). If that matters for this issue, reopen against a deployed check.

    Closing by hand: Closes #N does not fire here, since the PR targeted Dev_new_gui rather than the default branch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions