Skip to content

feat(llc/scheduler): registry adapters (claude_code) get no ProviderRateLimited backoff recovery #9773

Description

@mrveiss

Problem

The heartbeat scheduler's ProviderRateLimited exponential-backoff recovery (GH#8204) only fires when an adapter raises ProviderRateLimited — which happens for in-process autobot_agent agents via AutoBotAgentAdapter.run_blocking.

Registry adapters (e.g. claude_code) report status through the AdapterRunStatus dataclass (invoke/status/cancel), which has no rate-limited representation. When a claude_code subprocess hits a provider rate limit, it exits and surfaces as a non-success terminal status — now recorded as a failed run (PR #9772, AdapterRunFailed) rather than triggering backoff + work-item-checkout preservation.

So registry-adapter agents get no automatic rate-limit recovery; a rate-limited run is treated as a plain failure.

Proposed fix

  • Add a RATE_LIMITED channel to AdapterRunStatus (or a dedicated detector) so ClaudeCodeAdapter.status / subprocess output parsing can report rate limiting.
  • In _dispatch_registry_adapter / _await_adapter_completion, translate a RATE_LIMITED terminal status into a raised ProviderRateLimited so the existing backoff path in _run_adapter applies uniformly.

References

Discovered in code review of PR #9772 (#9622 / #9623). See _dispatch_registry_adapter in autobot-backend/llc/scheduler/heartbeat_scheduler.py.

Labels: enhancement · backend · agents · priority: medium

Activity

  1. mrveiss commented on Jun 9, 2026

    @mrveiss
    OwnerAuthor

    Related cleanup surfaced in review of #9835: claude_code_subscription_adapter.py and copilot_subscription_adapter.py each carry a # TODO: Implement auto-pause + board notification (depends on GH#8225) on the quota-exhaustion path — violates the repo "no TODO comments" rule. The auto-pause/board-notify behavior belongs to this issue's domain (quota/rate-limit handling for registry adapters); fold it in when implementing the RATE_LIMITED channel here.

  2. mrveiss commented on Jun 9, 2026

    @mrveiss
    OwnerAuthor

    Consolidation to fold in here (verified duplication): _check_quota_exhaustion() is byte-identical (bar log strings) between claude_code_subscription_adapter.py:210 and copilot_subscription_adapter.py:186, and each carries its own _QUOTA_EXHAUSTED_PATTERNS list + an identical quota-check block in status(). When implementing the RATE_LIMITED channel here, hoist _check_quota_exhaustion (and a per-adapter QUOTA_PATTERNS class attr) into a shared mixin / subprocess_support helper so the detection logic lives once. (codex_subscription_adapter is a NotImplementedError stub — no quota logic.)

  3. github-actions commented on Jun 11, 2026

    @github-actions
    Contributor

    PR #9993 (merged to Dev_new_gui) references this issue with a close keyword.

    feat(llc/scheduler): rate-limit detection + backoff for registry adapters (#9773)

    If this issue is fully resolved, close it manually. If work remains, no action is needed.

  4. mrveiss commented on Jun 11, 2026

    @mrveiss
    OwnerAuthor

    Closed — merged to Dev_new_gui via PR #9993 (squash commit a2be8d76e).

    Criteria met:

    • Shared rate-limit detection in subprocess_support (_RL_KEYWORDS, is_rate_limit_output, read_output_tail, final_result_event) — extracted from autobot_agent_adapter, which now delegates (also resolves this issue's _check_quota_exhaustion-dup note: subscription adapter's quota check now wins over inherited RATE_LIMITED, tested)
    • ClaudeCodeAdapter._status() reclassifies exited runs to RATE_LIMITED only when the stream-json terminal result event is missing (mid-stream kill) or carries an error — a successful run mentioning 'rate limit'/'429'/'quota' can never be re-queued (review-caught critical, covered by 3 dedicated false-positive tests)
    • _dispatch_registry_adapter translates terminal RATE_LIMITED → raise ProviderRateLimited — identical GH#8204 path (checkout preservation, exponential backoff, retry cap, rate_limited run record, no AdapterRunFailed); reviewer-verified parity. RATE_LIMITED never persisted for registry adapters → zero frontend ripple
    • 150 tests pass (21 in the new rate-limit suite + full adapter package)

    Discoveries: #9992 (claude CLI stderr PIPE'd, never drained — limits detector recall + potential pipe-buffer block).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions