Skip to content

fix(codex): classify spent subscription windows as quota rejections - #434

Merged
MagicalTux merged 3 commits into
KarpelesLab:masterfrom
rikbrown:fix/codex-spent-window-is-a-rejection
Sep 19, 2026
Merged

MagicalTux merged 3 commits into
KarpelesLab:masterfrom
rikbrown:fix/codex-spent-window-is-a-rejection

Conversation

@rikbrown

@rikbrown rikbrown commented Sep 19, 2026 •

Copy link
Copy Markdown
Contributor

The gap

forwardRequest distinguishes:

  • Quota rejection — durable: hold the account for its reset window and rotate the request.
  • Rate-limit throttle — momentary: pause the account, wait retry-after, and resend to the same account.

Quota rejection currently requires anthropic-ratelimit-unified-5h-status === 'rejected' or its 7d equivalent. The ChatGPT backend does not set these headers. It reports percentage quota in x-codex-*, which src/codex-quota.js already parses, so every Codex 429 entered the throttle path even when its subscription was spent.

With no sibling, the proxy slept a retry-after below the inline-absorb cap and resent to the exhausted subscription. With an idle sibling, the request could succeed after the single failover hop (#137), but the account stayed active behind a silent pause: the TUI showed it healthy, the throttledAt revalidation probe did not arm, and pauseAccount trained the concurrency learner from a spent quota.

The change

codexSpentWindows(headers) now reads the response through parseCodexQuota, the parser used by the quota sweep, and returns labels for windows at or past their limit. generalRejected tests whether the returned array has entries.

The parser groups x-codex-* families and uses each window's window-minutes, so it sees named model-scoped 5-hour windows that account-wide primary/secondary cannot. A spent session window can report flat values of 42% / 0% while refusing every request. classify also discards unparseable percentages and zero-length windows, which mean "not applicable".

updateQuota has already stored utilisation from these headers. The existing rejection branch then holds the account for retry-after and rotates. The log now includes the window, for example: Quota rejection (429) on "codex-1" (5h spent) — throttling 45s and switching account.

parseCodexQuota adds its missing @returns typedef, and rateLimitHeaders gets the Record<string, string> annotation now passed to it.

Hold semantics

The hold remains retry-after, clamped to [1s, 3600s], as in the Anthropic path. It does not use the window reset: retry-after is upstream's requested quiet period, while the recorded utilisation over the switch threshold keeps selection off the account for the rest of the window. Tests pin a 45s retry-after with a reset six hours out.

Verification

  • npm test — 1871 pass, 0 fail, including six new test/codex-quota-rejection.test.js cases. They cover account-wide and named-family windows, non-cases, and end-to-end createProxyServer requests to a local Codex-429 upstream.
  • Removing the new clause makes all three end-to-end tests fail on master: the two-account case leaves the account active and paused, and the lone-account cases wait 45s and call the exhausted subscription twice.
  • npm run lint, npm run typecheck, and node scripts/typecheck-strict.mjs are clean; the ratchet reports 6 fewer strict diagnostics than base.
  • Tests use no real endpoint or wall-clock time; they distinguish the paths by account state and upstream hit count.

Deliberate choice

A named family's weekly bucket is model-scoped, like Anthropic 7d_oi, which the existing branch treats as a Fable-only rejection without an account hold. This change treats it as general rejection. For now, upstream has refused this request on this account, the hold is bounded by retry-after, and updateQuota has recorded the bucket. codexSpentWindows labels already distinguish the case if a narrower model-scoped rejection is needed later.

🤖 Generated with Claude Code

…hrottle

A 429 is classified two ways here, and the difference decides everything that
follows. A quota rejection is durable: the account cannot serve again until a
window resets, so it is held and the request rotates. A rate-limit throttle is
momentary: the account is paused, the request waits out retry-after and goes
back to the SAME account.

That classification read only `anthropic-ratelimit-unified-*-status`. The
ChatGPT backend never sets it. It reports how much of each window is gone
instead, and a window at its limit is the same fact in the other spelling. So
every Codex 429 fell through to the throttle path. With no sibling to fail over
to, the proxy slept out retry-after and asked the exhausted subscription again,
to be refused again. With one, the single failover hop still carried the
request — but the account was left `active` behind a silent pause: nothing
recorded that it was out of service or until when, the revalidation probe that
keys off `throttledAt` never armed, and the pause taught the concurrency learner
that the account had choked at that load, which a spent quota is no evidence of.

Read the refusal through `parseCodexQuota` — the parser the quota sweep already
uses — rather than off the raw headers. It groups the `x-codex-*` families and
classifies each window by its own `window-minutes`, so `codexSpentWindows` sees
what the account-wide pair cannot: a subscription states its only 5-hour window
inside a NAMED, model-scoped family, and `x-codex-primary/secondary-used-percent`
alone would read 42% / 0% while the account was refusing on a spent session
window. The labels it returns name what is spent, and the rejection log line now
says which window it was.

Two annotations come with it: `parseCodexQuota` gets the `@returns` typedef it
never had, so a caller can read `modelBuckets` off it, and `rateLimitHeaders`
gets the `Record<string, string>` it is now passed as.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@rikbrown rikbrown changed the title fix(codex): a spent subscription window is a quota rejection, not a throttle fix(codex): classify spent subscription windows as quota rejections Sep 19, 2026
MagicalTux and others added 2 commits September 20, 2026 08:32
…ucket

A spent named-family weekly bucket was classified as a general rejection, which
parks the whole account for every model. Nothing in selection reads the recorded
model buckets, so once the hold lapsed the account was picked again, refused
again and was parked again. Only the account-wide `5h` and `weekly` windows now
hold the account; a rejection where only named-family windows are spent rotates
this request to another account, the way the Fable-only rejection does.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@MagicalTux
MagicalTux merged commit 2188705 into KarpelesLab:master Sep 19, 2026
5 checks passed
@MagicalTux MagicalTux mentioned this pull request Sep 20, 2026
MagicalTux added a commit that referenced this pull request Sep 20, 2026
Thirty-six commits since 1.1.20. Several change what a client sees on a
failure, so read the first section before upgrading a shared deployment.

Behaviour changes
  #439 a 401 from upstream is no longer relayed to the client: the request
       fails over like a 403, and an API-key account or an OAuth account with
       no refresh token leaves rotation (it used to be picked again on every
       request). With nothing left the client gets the proxy's own error
  #439 a path under /teamclaude/ that no control route claims — a typo, or the
       wrong verb — answers 404 locally instead of being forwarded upstream
       under a fleet credential
  #429 the synthetic 429's retry-after is the real reset of the windows
       blocking the request's candidate accounts, not a flat 60s, and the
       message counts only those candidates; #408 names accounts that need a
       re-login instead of calling them "at quota"
  #438 session pins are per conversation (session id plus a digest of the
       first message), so a session's subagents spread across accounts.
       `sessions.items[].id` in status is the composite key, load is counted
       per conversation, and a persisted concurrency cap re-learns
  #378 a `thread: continue` bound for a per-account third-party upstream is
       refused with the 400 Anthropic gives, so the client resends the whole
       conversation; `messageThreads: true` opts a relay out
  #434 a 429 whose x-codex-* headers show a spent account-wide window holds
       the account like an Anthropic rejection; a spent model-scoped bucket
       only moves the request
  #437 a Codex response head is awaited for five minutes (Anthropic unchanged)
  #411 idle keep-alive connections are held 120s on both listeners
  #389 with session distribution on, requests carrying no session id rotate on
       a cursor of their own instead of all resting on the current account
  #405 `defaultClientMode` ("mitm" | "base-url") sets what `run` and `env` do
       without a flag; `--mitm` / `--no-mitm` decide per launch, and in
       base-URL mode `env` unsets a stale proxy export naming this proxy
  #439 route `--bucket` is validated; an array `switchThreshold` reads as the
       default with one line saying so

Features
  #419 an MCP management endpoint at POST /teamclaude/mcp, off unless
       `proxy.mcp` is "read" or "full"; a named client key is read-only even
       in full mode, and with no proxy key it serves only this machine
  #428 per-account `switchThreshold`, a number or a per-bucket table
  #406 a Claude+Codex pool is drawn as two panes on a wide terminal, each with
       its own current marker; #392 names the provider in a mixed list; #418
       lets the operator arrange the list (`displayOrder`); #376 draws
       loopback-served accounts last; #435 shows the percentage beside a bar's
       countdown; #394 shows the running version in the header
  #430 free Codex rate-limit reset credits in status, the TUI and the dashboard
  #385 `proxy.terminalOnly` tunnels chatgpt.com so ChatGPT Desktop stays out

Fixes
  #404 a TUI paint can no longer block the proxy (stdout non-blocking, frames
       dropped while the terminal is behind); #410 a dead terminal no longer
       takes the proxy with it, and SIGHUP shuts down cleanly
  #433 token usage is booked from Codex Responses streams
  #386 #387 #388 the Codex five-hour window is read from the model-scoped
       family and the usage probe, and extra limits are named from their entries
  #431 a headerless 429 that follows the request is retried once
  #432 #439 startup and collaborator log lines reach the TUI's activity pane
       and log file instead of the covered terminal
  #415 #403 #439 reload mirrors `priority`, `disabled`, `stripRequestFields`
       onto the config entry, and a reload during a removal does not re-add it
  #439 the Host check uses the address actually bound; sx.org calls time out
  #381 the dashboard polls status before asking for a key
  #403 session outcome accounting classifies the decoded path; account names in
       daemon log lines are sanitised

Tooling
  #371 #372 #373 `npm run typecheck` (tsc over the JS sources) in CI, with a
       strict-mode ratchet: per-file strict diagnostics may not grow past the
       pre-merge commit (2006 at introduction, 1735 now)
  #401 docker workflow actions bumped

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
rikbrown added a commit to rikbrown/teamclaude that referenced this pull request Oct 7, 2026
…indows

Upstream KarpelesLab#434 classifies a spent Codex window through codexSpentWindows,
which reads each window by its own `window-minutes` and covers named
families. The sidecar's flat reading of the primary/secondary percentages
is gone with it, along with its tests; a forwarded `x-codex-*` header
reaches the upstream classifier exactly as a direct one does.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants