Skip to content

llm: route on observed headroom before dispatch instead of on a 429 after it; reconcile guessed RPM buckets with observed limits #15028

Description

@mrveiss

Parent: #15021

Problem

Fallback fires after the damage. model_fallback_coordinator.py switches models only once a
RateLimitError survives backoff retries (rate_limit_backoff.py → RateLimitHandler, max 5
retries, up to 120 s delay). So the cheapest possible outcome of hitting a limit is one wasted
round-trip plus a backoff sleep; the expensive outcome is five of them.

Meanwhile cross_worker_rate_limiter.py — the one proactive component — throttles against numbers
that are guesses. _provider_limits() (:118-126) reads AUTOBOT_LLM_RL_{PROVIDER}_RPM with
"conservative approximations of free-tier limits" as defaults, documented as something "operators
should override for their actual provider plan". In practice nobody does, so the bucket is either
throttling traffic the provider would have accepted, or waving through traffic it will reject.

#15026 makes the real numbers available. This task uses them.

Scope

  • llm_shared/model_fallback_coordinator.py — consult the headroom store before dispatch. When
    the target model's window is below a configurable floor and a fallback has room, switch
    pre-emptively and record it in LLMResponse.provider_metadata the same way reactive fallbacks are
    recorded today.
  • llm_shared/cross_worker_rate_limiter.py — when observed limits exist for a provider, they take
    precedence over the env-guessed defaults. The bucket becomes self-correcting; the env vars remain
    the bootstrap and the override.
  • llm_shared/provider_registry.py — skip providers in needs_reauth (llm: degradation store cannot distinguish a dead credential from a transient failure — dead tokens retry every TTL forever #15022) at selection time.
  • Emit the pre-emptive switch through the existing fallback_events path so
    ProviderFallbackView.vue / ProviderFallbackChip.vue surface it with no frontend change.
  • The headroom floor is a module-level constant from an env var.

Acceptance criteria

  • With headroom data present, a request targeting a near-exhausted window is routed to a fallback
    without a 429 first — proven by a test asserting zero rate-limited round-trips.
  • With no headroom data, behaviour is byte-for-byte today's reactive path. No regression.
  • Observed limits override env-guessed RPM only when fresher than a bounded age; a stale
    observation never widens the bucket.
  • A pre-emptive switch is auditable: it appears in provider_metadata and emits a fallback event,
    distinguishable from a reactive switch.
  • Providers in needs_reauth are never selected.
  • No routing loop: a chain where every member is near-exhausted degrades to the current behaviour
    rather than cycling.
  • Tests: pre-emptive switch, no-data passthrough, stale-observation rejection, all-exhausted
    chain, needs_reauth skip.

Implementation Order

Wave: 2
Depends on: #15026, #15022
Unblocks: #15021 Task 5

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions