feat: per-route "skip on quota exhaustion" toggle for subscription providers - #2739
feat: per-route "skip on quota exhaustion" toggle for subscription providers#2739tunwinlat wants to merge 8 commits into
Conversation
…n providers Subscription providers that expose a usage/quota endpoint (Anthropic, Kimi, OpenAI, MiniMax, xAI) can now be polled on an interval, and any route in a harness's routing (tier overrides, fallbacks, specificity and header-tier routes) can opt into being skipped while its connection's quota is exhausted. This stops the router from wasting attempts on models that are guaranteed to fail and gets to a working fallback faster. - SubscriptionQuotaService polls each active subscription connection every 60s (SUBSCRIPTION_QUOTA_POLL_INTERVAL_MS, min 30s), fail-open on any error, in-memory state only - ModelRoute gains an optional skipWhenQuotaExhausted flag (jsonb, no migration); honored for primaries and fallbacks alike, with a never-empty safeguard that keeps the original chain when every candidate is exhausted - Routing UI exposes the toggle as a gauge icon on quota-capable subscription routes
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #2739 +/- ##
==========================================
+ Coverage 96.48% 96.50% +0.02%
==========================================
Files 600 605 +5
Lines 35708 36385 +677
Branches 9977 10184 +207
==========================================
+ Hits 34452 35114 +662
- Misses 695 706 +11
- Partials 561 565 +4
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
Bundle ReportChanges will increase total bundle size by 18.82kB (1.2%) ⬆️. This is within the configured threshold ✅ Detailed changes
Affected Assets, Files, and Routes:view changes for bundle: manifest-frontend-esmAssets Changed:
Files in
Files in
|
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
All reported issues were addressed across 42 files
Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.
Re-trigger cubic
Cover the quota-skip toggle handlers and prop threading across Routing, RoutingActions, the tier/specificity/header-tier sections, HeaderTierCard, and the FallbackList revert path.
- Kimi parser: NaN sentinel for missing/corrupt quota numbers so absent data can never read as exhausted (fail-open) - setOverride (tier/specificity/header-tier): preserve a persisted skipWhenQuotaExhausted when the field is omitted from the PUT; explicit false still clears - legacy route conversion helpers round-trip the flag - quota toggles: in-flight guards against double-click races on fallback rows and header-tier primary chips - prune quota state for deleted/deactivated connections each poll - validate SUBSCRIPTION_QUOTA_POLL_INTERVAL_MS as digits-only - drop a dead isManual() guard on the primary chip toggle
There was a problem hiding this comment.
All reported issues were addressed across 17 files (changes from recent commits).
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
There was a problem hiding this comment.
All reported issues were addressed across 6 files (changes from recent commits).
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
|
hi @tunwinlat, thanks for your contribution! before moving forward, Manifest already has a cooldown system where it'll skip the route : 15s default for rate limits and 5 min max based on the Retry-After value in the error response. it's not enough for you? |
Hi @guillaumegay13, thanks for the attention. Sorry, I missed some contexts in the summary. I am trying to fix 2 things that I find a bit bothersome with retrying when quota is full on top of just failed attempts.
Please let me know if you have any questions or suggestions. |
There was a problem hiding this comment.
1 issue found across 7 files (changes from recent commits).
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="packages/frontend/src/pages/RoutingTierCard.tsx">
<violation number="1" location="packages/frontend/src/pages/RoutingTierCard.tsx:272">
P2: When a fallback is promoted in a specificity tier, `skipWhenQuotaExhausted` is passed here but the specificity `onOverride` chain ignores it, so the new primary route loses its quota-skip flag. Thread the optional flag through `RoutingSpecificitySection` and `handleSpecificityOverride` into `overrideSpecificity` (the same applies to the other swap branch).</violation>
</file>
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
| provId ?? '', | ||
| newPrimaryRoute?.authType, | ||
| newPrimaryRoute?.keyLabel ?? undefined, | ||
| skipWhenQuotaExhausted, |
There was a problem hiding this comment.
P2: When a fallback is promoted in a specificity tier, skipWhenQuotaExhausted is passed here but the specificity onOverride chain ignores it, so the new primary route loses its quota-skip flag. Thread the optional flag through RoutingSpecificitySection and handleSpecificityOverride into overrideSpecificity (the same applies to the other swap branch).
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/frontend/src/pages/RoutingTierCard.tsx, line 272:
<comment>When a fallback is promoted in a specificity tier, `skipWhenQuotaExhausted` is passed here but the specificity `onOverride` chain ignores it, so the new primary route loses its quota-skip flag. Thread the optional flag through `RoutingSpecificitySection` and `handleSpecificityOverride` into `overrideSpecificity` (the same applies to the other swap branch).</comment>
<file context>
@@ -252,13 +253,25 @@ const RoutingTierCard: Component<RoutingTierCardProps> = (props) => {
+ provId ?? '',
+ newPrimaryRoute?.authType,
+ newPrimaryRoute?.keyLabel ?? undefined,
+ skipWhenQuotaExhausted,
+ );
+ }
</file context>
2 participants
Summary
Subscription providers with usage windows (Anthropic, Kimi, OpenAI, MiniMax, xAI) currently keep getting attempted on every request even when their quota is exhausted — each attempt fails with a quota/rate-limit error before the fallback chain engages, adding avoidable latency to every routed request.
This PR adds an opt-in per-route "Skip on quota exhaustion" flag: a new backend service polls each subscription connection's usage endpoint on an interval, and the router skips flagged routes while their connection is known to be exhausted.
What changed
ModelRoutegainsskipWhenQuotaExhausted?: boolean(shared type; stored in the existing jsonb route columns — no migration). AquotaCheckcapability on the shared subscription configs marks which providers expose a usable usage endpoint.SubscriptionQuotaService(routing/subscription-quota.service.ts): polls every 60s (SUBSCRIPTION_QUOTA_POLL_INTERVAL_MS, clamped to ≥30s), in-memory state only, fail-open everywhere — fetch errors, unwrap failures, or missing data never skip a route, and a failed refresh keeps the previous state. Per provider:GET api.anthropic.com/api/oauth/usage(5h/weekly/per-model windows,utilization >= 100)GET api.kimi.com/coding/v1/usages(5h/weekly/monthly,used >= limit)GET chatgpt.com/backend-api/wham/usage(used_percent >= 100)GET <resource-origin>/v1/token_plan/remains(remaining-percent<= 0)POST grok.com/grok_api_v2.GrokBuildBilling/GetGrokCreditsConfig(gRPC-web + dependency-free protobuf scan)ResolveServiceskips flagged routes backed by an exhausted connection in both promotion paths (tier chain and header tiers), for primaries and fallbacks alike. A never-empty safeguard keeps the original chain when every candidate is exhausted, so a harness is never left unroutable.For users
New "Skip on quota exhaustion" toggle on subscription routes in a harness's routing. When enabled, an exhausted connection's models are bypassed until the quota window resets, so requests reach a working fallback immediately.
For operators
SUBSCRIPTION_QUOTA_POLL_INTERVAL_MS(default 60000, min 30000).Notes
/coding/v1/usagesis undocumented (reverse-engineered, used by several community tools); the parser is deliberately defensive (string numbers, window fallbacks).subscription-quota.*, notsubscription-usage.*) to avoid conflicting with the unmerged PR — the fetchers could be unified later.supportsQuotaCheck(shared configs, drives UI visibility) and the backend service's provider list are two sources of truth to keep in sync when adding providers.Validation
npm test --workspace=packages/shared— 429/429npm test --workspace=packages/backend— full suite 458/458 suites, 8413 testsnpm run build— clean; eslint clean on all changed filesSummary by cubic
Skips exhausted subscription routes during routing so requests hit working fallbacks faster. Previously every candidate was attempted even when its subscription was exhausted; now quota-capable routes can opt in to be skipped, with a safeguard that keeps chains non-empty.
SubscriptionQuotaServiceviaSubscriptionQuotaModule) that polls Anthropic, Moonshot (Kimi), OpenAI, MiniMax, and xAI every 60s (SUBSCRIPTION_QUOTA_POLL_INTERVAL_MS, min 30s; digits-only). In-memory, fail-open; failed refresh keeps prior state; prunes deleted/disabled connections; reuses existing OAuth refresh.manifest-sharedexposessupportsQuotaCheckand marks provider coverage. Docs and tests updated; no new runtime dependencies.Written for commit ec6f4ef. Summary will update on new commits.