Skip to content

agent_presence_sync hands an un-awaited coroutine to the agent terminal service, poisoning a process-wide singleton #17436

Closed

Description

@mrveiss

initialization/agent_presence_sync.py:79 hands a coroutine object to the agent terminal service as its Redis client:

terminal_service = get_agent_terminal_service(redis_client=get_redis_client(async_client=True))
RuntimeWarning: coroutine 'RedisConnectionManager.get_async_client' was never awaited
  autobot-backend/initialization/agent_presence_sync.py:79

Observed on merged main (5dcdb18cec), backend startup, run 36055943167 job 107824575073.

Three things make this worse than a stray warning

1. It poisons a process-wide singleton, and startup order decides it.

def ensure_agent_terminal_service(**kwargs: Any) -> AgentTerminalService:
    """The singleton, built from ``kwargs`` by whichever caller gets there first (thread-safe)."""

Whoever constructs it first fixes its redis_client for the life of the process. This call runs during startup presence sync, so it is a strong candidate for "first". Every later consumer that correctly awaits its own client still receives the singleton built from this one.

2. A coroutine is truthy, so the guard that should catch it does not.

services/agent_terminal/session_manager.py:285

if self.redis_client:
    await self._persist_session(session)

redis_client is a coroutine object, which passes the truthiness test, so _persist_session runs against it. The guard reads as "only persist if we have Redis" and actually means "only persist if we have something".

3. The form used is one the codebase already documents as a trap.

autobot_shared/redis_client.py:68 — "get_redis_client(async_client=True) is error-prone because it is a sync function that returns a coroutine" — and :303 names get_async_redis_client as "the safe alternative", added for #3962 which was this exact fault.

Census: this is the only production site that gets it wrong

Every other call of the unsafe form awaits it correctly:

Site Awaited?
autobot-backend/initialization/agent_presence_sync.py:79 no
autobot-backend/api/ws_security.py:417 yes
autobot-backend/services/agent_terminal/conversation_owner.py:48 yes

So this is one missed await, not a pattern — the value of the census is establishing that, so the fix is not over-scoped.

Also worth noting: a FastAPI dependency invoked outside a request

get_agent_terminal_service(redis_client=Depends(get_redis_client)) is a dependency function. Called directly from startup code, Depends(...) is inert and the caller must supply the argument by hand — which is exactly where this went wrong. A dependency called manually loses the framework's wiring and keeps its signature, so the mistake looks like ordinary keyword passing.

Acceptance criteria

  • The terminal service receives an awaited client. Prefer await get_async_redis_client(...) over awaiting the legacy form, since the codebase already names it the safe alternative.
  • session_manager.py:285's guard rejects a non-client. Truthiness is not a type check — a coroutine, a Mock, and a partially-built client all pass it.
  • A test fails against current code: construct the service the way agent_presence_sync does and assert persistence actually reaches Redis, not merely that no warning is emitted. A RuntimeWarning assertion passes as soon as someone adds an await anywhere, including the wrong one.
  • Verified that the singleton built at startup is the one later consumers get — or that it is not a singleton. Either resolves the ordering hazard; assuming it away does not.

Relationship to #16499

Same consequence, different call, different module. #16499 is scoped to EdgeLearner receiving an un-awaited get_async_redis_client() from neural_mesh_wiring.py. This is the agent terminal service receiving an un-awaited get_redis_client(async_client=True) from agent_presence_sync.py.

Filed separately because #16499's scope is EdgeLearner rather than the class. They should be fixed in one PR — neither is large, both are one missed await, and the standing rule is that one defect found twice is fixed once. Whoever takes either should take both.

Activity

  1. mrveiss commented on Sep 25, 2026

    @mrveiss
    OwnerAuthor

    The fix cannot be another warning — that mitigation has now failed three times

    Adding this as a constraint on the acceptance criteria, because the obvious response to "a caller forgot to await" is to document it better, and that has already been tried.

    redis_client.py:68 warns that get_redis_client(async_client=True) is error-prone. :303 names get_async_redis_client as the safe alternative. Both were added for #3962 — this same fault. Since then: #16499 (EdgeLearner) and this one. A docstring asking callers to remember is a mitigation with three recorded failures.

    So the criterion is that the trap cannot be fallen into, not that it is better signposted. Concretely, one of:

    • get_redis_client(async_client=True) stops returning a bare coroutine — it either returns an awaitable wrapper that fails loudly when used unawaited, or the sync function refuses async_client=True outright and directs callers to get_async_redis_client;
    • or the consumers validate. Not for truthiness — for being a client.

    That second half is the same class of silent pass and worth stating separately:

    if self.redis_client:      # session_manager.py:285

    A coroutine is truthy. So is a Mock, a half-built client, and a functools.partial. This guard reads as a client check and is a presence check, and a presence check will keep catching nobody — it passes for every wrong value anyone has actually passed. hasattr(client, "hgetall") or an isinstance against the redis type distinguishes what the guard is named after.

    Same shape as the DATABASE_MAPPING.get(name, 0) default in #17435: a branch that accepts the malformed input and continues, so the mistake reaches production wearing a green log line. Two different files, one habit — prefer failing on the bad value over having a sensible behaviour for it.

    Suggested addition to the criteria

    • The unsafe call form cannot silently produce a coroutine-as-client. Verified by a test that fails against current code — call it the way agent_presence_sync.py:79 does and assert the failure is loud.
    • No guard in the affected paths accepts a non-client by truthiness. Verified by passing a coroutine and asserting the guard rejects it, not that no warning appears.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions