Command Code 429 rate_limit_exceeded exhausts retries on long muse-spark turns (v2.59.0)

Client or integration

Codex App

Area

Provider adapter

Summary

Long Codex App turns via commandcode/meta-muse-spark-1.3-contributor (reasoning high) repeatedly die with exceeded retry limit, last status: 429 Too Many Requests. The failing request reports errorCode=rate_limit_exceeded, closeReason=non_stream, upstreamError=Provider error 429: Upstream model provider is temporarily unavailable. Please try again in a moment. I expected a transient provider throttle to be retried with cooldown / Retry-After awareness instead of killing the whole workflow, forcing manual resume.

Related but distinct from #532 (Kiro USER_REQUEST_RATE_EXCEEDED with independent per-request retry budgets). This is Command Code + muse-spark, single explicit-provider route, no concurrent-account pool involved.

Reproduction

Start opencodex proxy/dashboard v2.59.0 on Windows, Codex App connected via localhost:10100.
Select commandcode/meta-muse-spark-1.3-contributor, reasoning_effort=high, explicit-provider routing.
Run a long multi-tool turn (blanche Go queue work: read files, run commands, edit/verify). The turn makes several sequential model + tool calls.
Observe intermittent failure: exceeded retry limit, last status: 429 Too Many Requests.
Open dashboard log detail for the failed request (example below). Retry manually after cooldown and it often succeeds.

Observed on 2026-09-19 ~23:08 KST (Asia/Seoul). Same pattern recurred across separate threads on the same night (including a thread doing only GitHub fetch calls), so it is not specific to one conversation.

Version

v2.59.0 (dashboard header shows opencodex v2.59.0)

Operating system

Windows 11 x64

Provider and model

commandcode / meta/muse-spark-1.3-contributor (requested as commandcode/meta-muse-spark-1.3-contributor, effectiveEffort=high)

Logs or error output

time: 2026-09-19 23:08:32 +09:00
request: ocx-c28be5ec8a5880f6adbc2cbec4779f52
conversation: 9751b9b507b784dda09d5d817898616e
model: meta/muse-spark-1.3-contributor
provider: commandcode (Command Code - API)
reasoning: high (reasoning_effort=high)
status: 429
durationMs: 2351
errorCode: rate_limit_exceeded
closeReason: non_stream
upstreamError: Provider error 429: Upstream model provider is temporarily unavailable. Please try again in a moment.
usageStatus: unreported
attempts: 1 attempt, adapter=openai-chat, status=429, durationMs=2291
routeKind: explicit-provider, selected commandcode/meta-muse-spark-1.3-contributor (explicit-provider-namespace)

Full redacted raw log for the failed request (secrets/tokens removed, account label kept truncated):

{
  "requestId": "ocx-c28be5ec8a5880f6adbc2cbec4779f52",
  "model": "meta/muse-spark-1.3-contributor",
  "provider": "commandcode",
  "inboundProtocol": "responses",
  "requestedEffort": "high",
  "effectiveEffort": "high",
  "status": 429,
  "durationMs": 2351,
  "errorCode": "rate_limit_exceeded",
  "closeReason": "non_stream",
  "upstreamError": "Provider error 429: Upstream model provider is temporarily unavailable. Please try again in a moment.",
  "usageStatus": "unreported",
  "routeKind": "explicit-provider"
}

Codex-side terminal errors seen in the thread:

exceeded retry limit, last status: 429 Too Many Requests
stream disconnected before completion: Provider stream error: upstream stream ended without a terminal signal ([DONE] or finish_reason) — possible truncation

Screenshots and supporting files

Dashboard log/detail screenshots available on request. The earlier screenshot in-discussion showed only healthy 200 rows for the same model/provider, so I am attaching the failing request detail above instead.

Redacted configuration

{
  "provider": "commandcode",
  "model": "meta/muse-spark-1.3-contributor",
  "stream": true,
  "routeKind": "explicit-provider"
}

Expected behavior

Distinguish transient provider 429 (temporarily unavailable / throttled) from hard quota exhaustion.
Honor Retry-After when present; otherwise apply shared cooldown before retrying the same account/model route.
Avoid burning the whole retry budget across rapid sequential tool-call turns; re-check cooldown after waking.
Surface the last upstream error verbatim when retries are exhausted.

Checks

I searched existing issues and documentation.
I removed secrets, tokens, account details, request credentials, and personal data.

The observed single attempt is consistent with the current documented policy, not evidence that the generic retry engine ignored a configured budget. retryOn429 is opt-in for key-auth providers, and the redacted configuration in this report does not include it; absent policy means fail fast. The Codex-side “exceeded retry limit” is the client exhausting its own retry loop after OCX returned the upstream 429.

A current workaround is a bounded provider policy such as:

"retryOn429": {
  "attempts": 3,
  "intervalMs": 5000,
  "maxIntervalMs": 60000,
  "respectRetryAfter": true
}

That covers pre-stream HTTP 429s on the Command Code openai-chat route. It must not replay a response after stream bytes or billable effects.

I am treating the remaining request as a provider-default enhancement: decide whether Command Code should ship a patient fallback when no explicit policy exists, similar to the narrowly documented OpenCode Go exception. If it does, an explicit retryOn429 including enabled:false must override it, every send must share the request-wide budget, and tests must cover missing/malformed Retry-After, cancellation during wait, and re-check after wake. Relabeling from bug to enhancement; keep open for that product decision.

Addressed in part on dev by #5266 (043aa435ff8f86095f55cbe08f74d45b9858da59). Keeping this open for the rest.

What landed: rate-limit and quota-exhausted are now separate causes with different resend decisions and different metric label values, so exhausting retries against a quota wall is distinguishable from an ordinary rate limit.

What remains: the shared cooldown and Retry-After handling this issue also asks for are routing behaviour and are not in that branch.

docs: record the ambiguous-resend gate and what the R4 remainders reach

Rework #4942 and #4989 into one ambiguous-resend gate with one grant …

The per-request resend budget landed on dev in 3a0718c does not close this one either.

The 429 leg appears in that change only as one participant in the composition — how many further sends it may spend, not whether a rate-limited turn should back off at all. The retry and cooldown policy this report is about is unchanged, so a long turn on a single key can still exhaust its retries the same way.

Remaining scope: cooldown and Retry-After awareness for a provider 429 on a single-key pool, and whether a shared cooldown belongs to the key, the account or the model. That is still open.

Triage: priority: P2 — Command Code 429 exhausts retries on long turns.

Criteria (P2): Medium: provider/client-specific bug with a workaround, bounded enhancement tied to a tracked issue, perf, or CI reliability.

Matched pull requests:

fix(oauth): honour a stated Retry-After in the generic account pool #5756 (related / partial) — fix(oauth): honour a stated Retry-After in the generic account pool [priority: P1]

Fixed on dev in #6066 (merge 7d8459388c, commit 4fda15d338). With no retryOn429 set, key-auth Command Code at its canonical endpoints now waits out a 429 on the same key: 6 replays, a 10 s interval, a 60 s cap, and Retry-After honored. This is the same fallback OpenCode Go has. An explicit retryOn429, including enabled: false, still wins, and OAuth rows are never replayed. The shared cross-request cooldown discussed in the thread is not part of this change.

fix(command-code): wait out a burst 429 on the canonical key endpoint

Navigation Menu

Repository navigation

Command Code 429 rate_limit_exceeded exhausts retries on long muse-spark turns (v2.59.0) #5180

Client or integration

Area

Summary

Reproduction

Version

Operating system

Provider and model

Logs or error output

Screenshots and supporting files

Redacted configuration

Expected behavior

Checks

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Command Code 429 rate_limit_exceeded exhausts retries on long muse-spark turns (v2.59.0) #5180

Description

Client or integration

Area

Summary

Reproduction

Version

Operating system

Provider and model

Logs or error output

Screenshots and supporting files

Redacted configuration

Expected behavior

Checks

Activity

Ingwannu commented on Sep 20, 2026

lidge-jun commented on Sep 20, 2026

lidge-jun commented on Sep 21, 2026

devin-ai-integration commented on Sep 24, 2026

lidge-jun commented on Sep 27, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions