You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Per-provider concurrency cap does not cover stream_completion() — only chat_completion() is bounded #16538
#16527 added a per-provider asyncio.Semaphore admission-control cap in BaseProvider.chat_completion() (autobot-backend/llm_shared/base_provider.py),
sized from LLMSettings.max_concurrent_requests. It only covers chat_completion(). stream_completion() is abstract on BaseProvider with
no shared wrapper — each of the 8 provider subclasses
(anthropic.py, vertexai.py, bedrock.py, vllm_base.py, huggingface.py, ollama_provider.py, mistral.py, openai_compatible.py) implements it
independently, calling its own network client directly. None of them go
through BaseProvider's breaker, rate limiter, backoff handler, or the new
concurrency semaphore.
services/llm_service.py::LLMService.stream() calls provider.stream_completion(request) directly (never chat_completion()),
so every streaming request bypasses all of that protection, not just the new
cap — this is a pre-existing gap the #16527 audit surfaced as a side effect,
not introduced by it.
Location
autobot-backend/llm_shared/base_provider.py — chat_completion()'s wrapper
(breaker, rate limiter, concurrency semaphore) has no streaming counterpart
8 provider-specific stream_completion() implementations under autobot-backend/llm_shared/providers/
Impact
A burst of streaming requests to one provider has none of: circuit-breaker
fail-fast, cross-worker rate limiting, backoff/retry, or (after #16527)
concurrency admission control. Only non-streaming chat() is protected today.
Acceptance Criteria
Decide and document the shared choke point for streaming requests (a
wrapper analogous to chat_completion()'s _attempt(), or a per-call
semaphore acquired in LLMService.stream() around provider.stream_completion())
The per-provider concurrency cap (LLMSettings.max_concurrent_requests)
is enforced for streaming requests too
Test demonstrates a streaming burst past the cap queues rather than
running unbounded
No existing streaming test broken
Context
Discovered while implementing #16527 (per-provider concurrency cap). Scoped
out of that PR: retrofitting 8 independent stream_completion()
implementations is a materially larger, separate-risk change from adding one
semaphore to the single chat_completion() call path. Related umbrella: #16524.
Witness, not a second defect (2026-10-02, while settling #16857): the same gap covers rate limiting, not only concurrency. LLMCrossWorkerRateLimiter (get_llm_rate_limiter()) is acquired at llm_shared/base_provider.py:182, inside the same block as the per-provider concurrency semaphore in chat_completion. stream_completion acquires neither. Checked in all four providers that implement it (openai_compatible, ollama_provider, mistral, bedrock): 0 references to the limiter. So the fix here should put the stream path under both the semaphore and the cross-worker rate limiter. #16857 deliberately does not wire a second LLM limiter, and its module comment points here for this gap.
Activity
mrveiss commentedon Oct 2, 2026
Witness, not a second defect (2026-10-02, while settling #16857): the same gap covers rate limiting, not only concurrency.
LLMCrossWorkerRateLimiter(get_llm_rate_limiter()) is acquired atllm_shared/base_provider.py:182, inside the same block as the per-provider concurrency semaphore inchat_completion.stream_completionacquires neither. Checked in all four providers that implement it (openai_compatible,ollama_provider,mistral,bedrock): 0 references to the limiter. So the fix here should put the stream path under both the semaphore and the cross-worker rate limiter. #16857 deliberately does not wire a second LLM limiter, and its module comment points here for this gap.fix(rate-limiting): enforce the per-user limit on chat sends, and rec…