Skip to content

Per-provider concurrency cap does not cover stream_completion() — only chat_completion() is bounded #16538

Open

Description

@mrveiss

Problem

#16527 added a per-provider asyncio.Semaphore admission-control cap in
BaseProvider.chat_completion() (autobot-backend/llm_shared/base_provider.py),
sized from LLMSettings.max_concurrent_requests. It only covers
chat_completion(). stream_completion() is abstract on BaseProvider with
no shared wrapper — each of the 8 provider subclasses
(anthropic.py, vertexai.py, bedrock.py, vllm_base.py, huggingface.py,
ollama_provider.py, mistral.py, openai_compatible.py) implements it
independently, calling its own network client directly. None of them go
through BaseProvider's breaker, rate limiter, backoff handler, or the new
concurrency semaphore.

services/llm_service.py::LLMService.stream() calls
provider.stream_completion(request) directly (never chat_completion()),
so every streaming request bypasses all of that protection, not just the new
cap — this is a pre-existing gap the #16527 audit surfaced as a side effect,
not introduced by it.

Location

  • autobot-backend/llm_shared/base_provider.py — chat_completion()'s wrapper
    (breaker, rate limiter, concurrency semaphore) has no streaming counterpart
  • autobot-backend/services/llm_service.py::LLMService.stream() — calls
    provider.stream_completion() directly
  • 8 provider-specific stream_completion() implementations under
    autobot-backend/llm_shared/providers/

Impact

A burst of streaming requests to one provider has none of: circuit-breaker
fail-fast, cross-worker rate limiting, backoff/retry, or (after #16527)
concurrency admission control. Only non-streaming chat() is protected today.

Acceptance Criteria

  • Decide and document the shared choke point for streaming requests (a
    wrapper analogous to chat_completion()'s _attempt(), or a per-call
    semaphore acquired in LLMService.stream() around
    provider.stream_completion())
  • The per-provider concurrency cap (LLMSettings.max_concurrent_requests)
    is enforced for streaming requests too
  • Test demonstrates a streaming burst past the cap queues rather than
    running unbounded
  • No existing streaming test broken

Context

Discovered while implementing #16527 (per-provider concurrency cap). Scoped
out of that PR: retrofitting 8 independent stream_completion()
implementations is a materially larger, separate-risk change from adding one
semaphore to the single chat_completion() call path. Related umbrella: #16524.

Activity

  1. added this to the v0.16.0 milestone on Sep 14, 2026
  2. mrveiss commented on Oct 2, 2026

    @mrveiss
    OwnerAuthor

    Witness, not a second defect (2026-10-02, while settling #16857): the same gap covers rate limiting, not only concurrency. LLMCrossWorkerRateLimiter (get_llm_rate_limiter()) is acquired at llm_shared/base_provider.py:182, inside the same block as the per-provider concurrency semaphore in chat_completion. stream_completion acquires neither. Checked in all four providers that implement it (openai_compatible, ollama_provider, mistral, bedrock): 0 references to the limiter. So the fix here should put the stream path under both the semaphore and the cross-worker rate limiter. #16857 deliberately does not wire a second LLM limiter, and its module comment points here for this gap.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions