feat(llm): quota-triggered model fallback (#8998)

[mrveiss]

"## Summary\n\nImplements automatic model fallback when primary model hits rate limits or quota caps (GH#8998).\n\nAddresses Paperclip #3317 and #2101 — AutoBot's complexity router now gracefully handles 429 errors by automatically trying fallback models instead of hard failures.\n\n## Thinking Path\n\nWhen a user's primary LLM model hits rate limits or quota caps, AutoBot should gracefully fall back to alternative models rather than returning errors. The approach:\n\n1. Created a centralized FallbackChainManager to define fallback sequences per model\n2. Integrated rate limit detection into LLMService.chat() to catch 429 errors\n3. On rate limit, query the manager for the next model in the chain and retry\n4. Track attempted models to prevent infinite loops (max 10 attempts)\n5. Support both environment variable and programmatic chain configuration\n\nThis allows operators to define custom fallback chains (e.g., opus → sonnet → haiku, or even cross-provider fallbacks).\n\n## What Changed\n\n### Core Implementation\n\n**llm_shared/fallback_chain.py\n- FallbackChain dataclass for defining fallback sequences\n- FallbackChainManager singleton with default chains:\n - Anthropic: claude-opus-4 → claude-sonnet-4 → claude-haiku-4\n - OpenAI: gpt-4 → gpt-4o → gpt-4o-mini\n - Cross-provider: claude-opus-4-cross → gpt-4o → ollama/llama3\n- Environment variable configuration: AUTOBOT_FALLBACK_CHAIN_<MODEL>=fallback1,fallback2\n- Provider-qualified chains: AUTOBOT_FALLBACK_CHAIN_OPUS=anthropic:claude-sonnet-4,openai:gpt-4o\n\nservices/llm_service.py\n- Modified LLMService.chat() to detect rate limit errors via extract_rate_limit_info()\n- On 429/quota error, queries FallbackChainManager for next fallback model\n- Retries with fallback model instead of failing immediately\n- Tracks attempted models to prevent infinite loops (max 10 fallback attempts)\n- Logs full fallback chain on success: \"Fallback successful: anthropic:claude-sonnet-4 worked after 2 attempts (tried: anthropic:claude-opus-4 → anthropic:claude-sonnet-4)\"\n\nllm_shared/__init__.py\n- Exported FallbackChain, FallbackChainManager, get_fallback_chain_manager\n\n### Tests\n\nllm_shared/fallback_chain_test.py\n- Full test coverage for FallbackChain dataclass\n- Default chain loading verification\n- Environment variable configuration tests\n- Cross-provider fallback validation\n- Singleton behavior tests\n\n## Verification\n\nRun tests with:\nbash\npytest autobot-backend/llm_shared/fallback_chain_test.py -v\n\n\nManual integration test:\npython\nfrom services.llm_service import get_llm_service\n\nsvc = get_llm_service()\nresponse = await svc.chat(\n messages=[{\"role\": \"user\", \"content\": \"Hello\"}],\n model_name=\"claude-opus-4\", # Will fall back to sonnet/haiku on 429\n)\n\n\nTest results:**\n- ✅ All unit tests pass (fallback_chain_test.py)\n- ✅ Default chains load correctly\n- ✅ Environment variable configuration works\n- ✅ Cross-provider fallback validation passes\n- ✅ Singleton behavior verified\n\n## Model Used\n\nClaude Sonnet 4.5\n\n---\n\n## How It Works\n\n1. User calls LLMService.chat() with primary model (e.g., claude-opus-4)\n2. Primary model returns 429 rate limit error\n3. LLMService checks FallbackChainManager for registered fallback chain\n4. If fallback exists, retries with next model in chain (e.g., claude-sonnet-4)\n5. Continues through fallback chain until success or exhaustion\n6. Logs full chain on success; returns error if all models exhausted\n\n## Configuration Examples\n\n### Environment Variables\n\nbash\n# Single-provider fallback\nexport AUTOBOT_FALLBACK_CHAIN_CLAUDE_OPUS=claude-sonnet-4,claude-haiku-4\n\n# Cross-provider fallback with explicit provider prefixes\nexport AUTOBOT_FALLBACK_CHAIN_MY_MODEL=anthropic:claude-sonnet-4,openai:gpt-4o,ollama:llama3\n\n# Simple model-only fallback (uses same provider)\nexport AUTOBOT_FALLBACK_CHAIN_GPT4=gpt-4o,gpt-4o-mini\n\n\n### Programmatic Registration\n\npython\nfrom llm_shared.fallback_chain import FallbackChain, get_fallback_chain_manager\n\nmanager = get_fallback_chain_manager()\nmanager.register_chain(FallbackChain(\n primary_model=\"my-custom-model\",\n fallback_models=[\"fallback-1\", \"fallback-2\"],\n primary_provider=\"openai\",\n fallback_providers=[\"openai\", \"ollama\"]\n))\n\n\n## Acceptance Criteria\n\n- ✅ FallbackChainManager with default chains for Anthropic and OpenAI\n- ✅ Environment variable configuration support\n- ✅ Cross-provider fallback chains\n- ✅ Integration with LLMService.chat()\n"

[github-actions]

✅ SSOT Configuration Compliance: Passing

🎉 No hardcoded values detected that have SSOT config equivalents!

[mrveiss]

✅ Ready to merge

Code Review:

• Well-structured fallback chain implementation
• Comprehensive test coverage (fallback_chain_test.py)
• Proper error handling and loop prevention (max 10 attempts)
• Clean integration with LLMService.chat()
• Supports env var config and cross-provider chains

CI Status:

• ✅ smoke-test: SUCCESS
• ✅ All security checks: SUCCESS
• ⚠️ code-quality/hardened-smoke-test failures are pre-existing project-wide issues (non-blocking)

No regressions identified. Merging now.