You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
llm: route on observed headroom before dispatch instead of on a 429 after it; reconcile guessed RPM buckets with observed limits #15028
Fallback fires after the damage. model_fallback_coordinator.py switches models only once a RateLimitError survives backoff retries (rate_limit_backoff.py → RateLimitHandler, max 5
retries, up to 120 s delay). So the cheapest possible outcome of hitting a limit is one wasted
round-trip plus a backoff sleep; the expensive outcome is five of them.
Meanwhile cross_worker_rate_limiter.py — the one proactive component — throttles against numbers
that are guesses. _provider_limits() (:118-126) reads AUTOBOT_LLM_RL_{PROVIDER}_RPM with
"conservative approximations of free-tier limits" as defaults, documented as something "operators
should override for their actual provider plan". In practice nobody does, so the bucket is either
throttling traffic the provider would have accepted, or waving through traffic it will reject.
#15026 makes the real numbers available. This task uses them.
Scope
llm_shared/model_fallback_coordinator.py — consult the headroom store before dispatch. When
the target model's window is below a configurable floor and a fallback has room, switch
pre-emptively and record it in LLMResponse.provider_metadata the same way reactive fallbacks are
recorded today.
llm_shared/cross_worker_rate_limiter.py — when observed limits exist for a provider, they take
precedence over the env-guessed defaults. The bucket becomes self-correcting; the env vars remain
the bootstrap and the override.
Emit the pre-emptive switch through the existing fallback_events path so ProviderFallbackView.vue / ProviderFallbackChip.vue surface it with no frontend change.
The headroom floor is a module-level constant from an env var.
Acceptance criteria
With headroom data present, a request targeting a near-exhausted window is routed to a fallback without a 429 first — proven by a test asserting zero rate-limited round-trips.
With no headroom data, behaviour is byte-for-byte today's reactive path. No regression.
Observed limits override env-guessed RPM only when fresher than a bounded age; a stale
observation never widens the bucket.
A pre-emptive switch is auditable: it appears in provider_metadata and emits a fallback event,
distinguishable from a reactive switch.
Providers in needs_reauth are never selected.
No routing loop: a chain where every member is near-exhausted degrades to the current behaviour
rather than cycling.
Parent: #15021
Problem
Fallback fires after the damage.
model_fallback_coordinator.pyswitches models only once aRateLimitErrorsurvives backoff retries (rate_limit_backoff.py→RateLimitHandler, max 5retries, up to 120 s delay). So the cheapest possible outcome of hitting a limit is one wasted
round-trip plus a backoff sleep; the expensive outcome is five of them.
Meanwhile
cross_worker_rate_limiter.py— the one proactive component — throttles against numbersthat are guesses.
_provider_limits()(:118-126) readsAUTOBOT_LLM_RL_{PROVIDER}_RPMwith"conservative approximations of free-tier limits" as defaults, documented as something "operators
should override for their actual provider plan". In practice nobody does, so the bucket is either
throttling traffic the provider would have accepted, or waving through traffic it will reject.
#15026 makes the real numbers available. This task uses them.
Scope
llm_shared/model_fallback_coordinator.py— consult the headroom store before dispatch. Whenthe target model's window is below a configurable floor and a fallback has room, switch
pre-emptively and record it in
LLMResponse.provider_metadatathe same way reactive fallbacks arerecorded today.
llm_shared/cross_worker_rate_limiter.py— when observed limits exist for a provider, they takeprecedence over the env-guessed defaults. The bucket becomes self-correcting; the env vars remain
the bootstrap and the override.
llm_shared/provider_registry.py— skip providers inneeds_reauth(llm: degradation store cannot distinguish a dead credential from a transient failure — dead tokens retry every TTL forever #15022) at selection time.fallback_eventspath soProviderFallbackView.vue/ProviderFallbackChip.vuesurface it with no frontend change.Acceptance criteria
without a 429 first — proven by a test asserting zero rate-limited round-trips.
observation never widens the bucket.
provider_metadataand emits a fallback event,distinguishable from a reactive switch.
needs_reauthare never selected.rather than cycling.
chain,
needs_reauthskip.Implementation Order
Wave: 2
Depends on: #15026, #15022
Unblocks: #15021 Task 5