Repository navigation
feat(llm): quota-triggered model fallback (GH#8998) - #9442
Merged
Merged
Conversation
…(GH#8998) Create ModelFallbackCoordinator that intercepts RateLimitError from BaseProvider.chat_completion() and automatically retries using the next model in the configured fallback chain before surfacing the error. Changes: - llm_shared/model_fallback_coordinator.py: new coordinator class with execute_with_fallback(), singleton accessor, audit metadata injection - llm_shared/__init__.py: export ModelFallbackCoordinator + get_fallback_coordinator - llm_multi_provider.py: wire coordinator into UnifiedLLMInterface.chat_completion() - api/openai_compat.py: wire coordinator into non-streaming dispatch path - api/anthropic_compat.py: same for Anthropic-compat endpoint - conftest.py: register real fallback modules for test collection - model_fallback_coordinator_test.py: 8 unit tests covering primary success, single fallback, multi-hop, exhaustion, no chain, disabled, audit metadata, and None provider Default chains (from fallback_chain.py): claude-opus-4 → claude-sonnet-4 → claude-haiku-4 gpt-4 → gpt-4o → gpt-4o-mini Cross-provider: claude-opus-4-cross → gpt-4o → ollama/llama3 Configurable via AUTOBOT_FALLBACK_CHAIN_<MODEL>=provider:fallback1,... Co-Authored-By: Paperclip <noreply@paperclip.ing>
Integrate ModelFallbackCoordinator into the actual dispatch paths: - llm_shared/__init__.py: export coordinator symbols - llm_multi_provider.py: UnifiedLLMInterface.chat_completion() - api/openai_compat.py: /v1/chat/completions non-streaming path - api/anthropic_compat.py: /v1/messages non-streaming path Non-streaming endpoints now automatically fall back to configured chain models on rate limit instead of surfacing 429 to caller. Co-Authored-By: Paperclip <noreply@paperclip.ing>
… Copilot (#9033) Implements GH#9033 — LLC agents can now run on Claude Max, GitHub Copilot, or ChatGPT Plus subscriptions without requiring separate API keys. **New Adapters:** - ClaudeCodeSubscriptionAdapter — Claude Code CLI subscription mode - Strips ANTHROPIC_API_KEY from environment - Parses quota exhaustion from CLI output - Auto-pause on quota exceeded (placeholder for scheduler integration) - CopilotSubscriptionAdapter — GitHub Copilot CLI subscription mode - Verifies gh auth before spawning - GitHub OAuth via gh auth session - Quota detection from CLI output - CodexSubscriptionAdapter — OpenAI Codex CLI stub - NotImplementedError (CLI doesn't exist yet) - Ready for future OpenAI CLI releases **Infrastructure:** - Added OAUTH_REFRESH_TOKEN to SecretType enum (models/secret.py) - Registered all 3 adapters in llc/adapters/__init__.py **Tests:** - 10 new tests in test_subscription_adapters.py (all passing) - Protocol conformance verified - Quota exhaustion detection validated - API key stripping verified **Integration Deferred:** - OAuth browser auth flow (needs auth service API) - Auto-pause scheduler integration (needs GH#8225) - Board notifications on quota events (needs notification system) Co-Authored-By: Paperclip <noreply@paperclip.ing>
… when approaching model context limit (#9043) Implements automatic context window management with rolling summarization: **Backend (api/chat.py + schemas_chat.py)**: - New POST /chat/summarize endpoint generates compact summaries of message ranges - ConversationSummarizeRequest/Data schemas for request/response - Uses gpt-4o-mini with temperature 0.3 for cost-effective summarization **Frontend (composables + components)**: - New useContextOverflowProtection composable monitors context usage - Auto-triggers summarization at 85% threshold (configurable) - Replaces oldest messages with summary to maintain rolling window - Enhanced ContextWindowIndicator shows 'Summarizing...' progress state - Integrated into ChatInterface with automatic message replacement **Behavior**: - Detection: builds on #8990 context tracking (80% warning, 85% auto-summarize) - Summarization: oldest ~40% of messages → compact summary - Target: reduce context to ~50% usage, giving room for new messages - Rolling window: preserves recent context while condensing history - Only active when persistHistory enabled Resolves GH#9043 Co-Authored-By: Paperclip <noreply@paperclip.ing>
- Resolved merge conflict in conftest.py (kept HEAD version with full dependency loading) - Reformatted 16 files with Black --line-length=120 per CI requirements Co-Authored-By: Paperclip <noreply@paperclip.ing>
Contributor
✅ SSOT Configuration Compliance: Passing🎉 No hardcoded values detected that have SSOT config equivalents! |
mrveiss
added a commit
that referenced
this pull request
Jun 4, 2026
* feat(llm): quota-triggered model fallback — auto-route on rate limit (GH#8998) Create ModelFallbackCoordinator that intercepts RateLimitError from BaseProvider.chat_completion() and automatically retries using the next model in the configured fallback chain before surfacing the error. Changes: - llm_shared/model_fallback_coordinator.py: new coordinator class with execute_with_fallback(), singleton accessor, audit metadata injection - llm_shared/__init__.py: export ModelFallbackCoordinator + get_fallback_coordinator - llm_multi_provider.py: wire coordinator into UnifiedLLMInterface.chat_completion() - api/openai_compat.py: wire coordinator into non-streaming dispatch path - api/anthropic_compat.py: same for Anthropic-compat endpoint - conftest.py: register real fallback modules for test collection - model_fallback_coordinator_test.py: 8 unit tests covering primary success, single fallback, multi-hop, exhaustion, no chain, disabled, audit metadata, and None provider Default chains (from fallback_chain.py): claude-opus-4 → claude-sonnet-4 → claude-haiku-4 gpt-4 → gpt-4o → gpt-4o-mini Cross-provider: claude-opus-4-cross → gpt-4o → ollama/llama3 Configurable via AUTOBOT_FALLBACK_CHAIN_<MODEL>=provider:fallback1,... Co-Authored-By: Paperclip <noreply@paperclip.ing> * feat(llm): wire fallback coordinator into call sites (GH#8998) Integrate ModelFallbackCoordinator into the actual dispatch paths: - llm_shared/__init__.py: export coordinator symbols - llm_multi_provider.py: UnifiedLLMInterface.chat_completion() - api/openai_compat.py: /v1/chat/completions non-streaming path - api/anthropic_compat.py: /v1/messages non-streaming path Non-streaming endpoints now automatically fall back to configured chain models on rate limit instead of surfacing 429 to caller. Co-Authored-By: Paperclip <noreply@paperclip.ing> * feat(llc/adapters): subscription-mode adapters for Claude Max, Codex, Copilot (#9033) Implements GH#9033 — LLC agents can now run on Claude Max, GitHub Copilot, or ChatGPT Plus subscriptions without requiring separate API keys. **New Adapters:** - ClaudeCodeSubscriptionAdapter — Claude Code CLI subscription mode - Strips ANTHROPIC_API_KEY from environment - Parses quota exhaustion from CLI output - Auto-pause on quota exceeded (placeholder for scheduler integration) - CopilotSubscriptionAdapter — GitHub Copilot CLI subscription mode - Verifies gh auth before spawning - GitHub OAuth via gh auth session - Quota detection from CLI output - CodexSubscriptionAdapter — OpenAI Codex CLI stub - NotImplementedError (CLI doesn't exist yet) - Ready for future OpenAI CLI releases **Infrastructure:** - Added OAUTH_REFRESH_TOKEN to SecretType enum (models/secret.py) - Registered all 3 adapters in llc/adapters/__init__.py **Tests:** - 10 new tests in test_subscription_adapters.py (all passing) - Protocol conformance verified - Quota exhaustion detection validated - API key stripping verified **Integration Deferred:** - OAuth browser auth flow (needs auth service API) - Auto-pause scheduler integration (needs GH#8225) - Board notifications on quota events (needs notification system) Co-Authored-By: Paperclip <noreply@paperclip.ing> * feat(chat): context overflow protection — auto-summarize conversation when approaching model context limit (#9043) Implements automatic context window management with rolling summarization: **Backend (api/chat.py + schemas_chat.py)**: - New POST /chat/summarize endpoint generates compact summaries of message ranges - ConversationSummarizeRequest/Data schemas for request/response - Uses gpt-4o-mini with temperature 0.3 for cost-effective summarization **Frontend (composables + components)**: - New useContextOverflowProtection composable monitors context usage - Auto-triggers summarization at 85% threshold (configurable) - Replaces oldest messages with summary to maintain rolling window - Enhanced ContextWindowIndicator shows 'Summarizing...' progress state - Integrated into ChatInterface with automatic message replacement **Behavior**: - Detection: builds on #8990 context tracking (80% warning, 85% auto-summarize) - Summarization: oldest ~40% of messages → compact summary - Target: reduce context to ~50% usage, giving room for new messages - Rolling window: preserves recent context while condensing history - Only active when persistHistory enabled Resolves GH#9043 Co-Authored-By: Paperclip <noreply@paperclip.ing> * fix(ci): resolve merge conflict and reformat with Black (MVA-3026) - Resolved merge conflict in conftest.py (kept HEAD version with full dependency loading) - Reformatted 16 files with Black --line-length=120 per CI requirements Co-Authored-By: Paperclip <noreply@paperclip.ing> --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements automatic model-level fallback when a primary model hits rate limit or quota cap (GH#8998).
When BaseProvider.chat_completion() exhausts retries on a RateLimitError, ModelFallbackCoordinator automatically tries the next model in the configured fallback chain before surfacing the error to the caller.
Changes
Core:
llm_shared/model_fallback_coordinator.py: new coordinator class with execute_with_fallback(), singleton accessor, audit metadata injectionllm_shared/model_fallback_coordinator_test.py: 8 unit tests (all pass)Integration:
llm_shared/__init__.py: export coordinator symbolsllm_multi_provider.py: wire into UnifiedLLMInterface.chat_completion()api/openai_compat.py: wire into /v1/chat/completions non-streaming pathapi/anthropic_compat.py: wire into /v1/messages non-streaming pathconftest.py: register real fallback modules for test collectionDefault Fallback Chains
From existing
fallback_chain.py:claude-opus-4 → claude-sonnet-4 → claude-haiku-4gpt-4 → gpt-4o → gpt-4o-miniclaude-opus-4-cross → gpt-4o → ollama/llama3Configuration
Environment variable:
Audit Trail
Fallback metadata injected into
LLMResponse.provider_metadata:fallback_used: boolprimary_model: strfallback_model: str (if fallback triggered)fallback_reason: "rate_limit_429"attempt_count: intfallback_chain_tried: list[str]Testing
pytest autobot-backend/llm_shared/model_fallback_coordinator_test.py # 8 passed in 0.44sResolves