Repository navigation
[Bug]: 2.41.0 keeps routing to a 5h-exhausted Codex account after repeated 502s #3425
Description
Activity
Issue reopened
The report now contains the information required by the automated check. Thanks for updating it.
리뷰 · 우선순위 70 / 80
설명
이 이슈는 OpenCodex 2.41.0에서 Codex 계정 풀 자동 전환이 켜져 있고(quota 전략, 임계 80%, sticky 1) 계정 A의 5시간 창이 100%인데도, 새 Codex App 요청이 A로 계속 붙어 HTTP 502를 내고, A를 수동 pause한 뒤에야 계정 B(3%)로 200이 난 사고 보고입니다. 로그 요약이 설득력 있습니다. A로 118번의 502,
sendCount전부 1,recoveryKinds비어 있음, 일부는streamAborted, 업스트림 본문은 저장되지 않음. 수동 pause 직후 B 성공. 관련 선례로 #3029(수동 pause 회복), #3110(풀 버스트 창), #3176(502에 싸인 quota 메시지 로테이션), #585(스트림 전 429/402 페일오버)를 스스로 연결했습니다.현재
dev(072df52eb, 패키지 2.42.0 라인) 기준으로 보면 “임계값 설정이 없다”는 문제는 아닙니다.src/codex/routing.ts의hasCodexQuotaHeadroom/applyQuotaAutoSwitch는autoSwitchThreshold(기본 80)와computeCodexUsageScore로 active를 넘기고,src/codex/quota.ts의isCodexQuotaExhausted는shortPercent100을 계획과 무관하게 소진으로 봅니다.pause-exhaustedAPI(src/codex/auth-api.ts)도 스냅샷이 100이면 pause합니다. 그런데 보고는 대시보드 quota refresh에는 A가 100으로 보이는데도, 새 admission이 A를 계속 고르고, 실패가 same-request recovery로도 안 넘어간 상태입니다. 이슈가 짚은 가설이 핵심입니다. (1) 스트림이 시작된 뒤의 실패는 터미널로 두고, (2) 본문 없는 502가 신선한 100% 스냅샷을 덮어쓰거나 전환을 막으면 안 되며, (3) affinity/sticky가 A에 묶인 채 다음 요청도 A로 가면 안 됩니다. 릴레이 쪽(src/server/relay.ts)은 post-200 스트림 실패를 synthetic 502 +streamAborted로 남기는 경로가 있어, “502만 보고 quota 로테이션”과 “이미 소진으로 알려진 계정은 다음 admission에서 제외”를 섞으면 구멍이 납니다.재현 제안(임계 80% + A short 100 + B 건강 + 동시/연속 새 요청, streamAborted는 터미널이어도 다음 admission은 A 회피, 본문 없는 502가 스냅샷을 이기지 못함, affinity 재평가)은 그대로 회귀 테스트 스펙으로 쓰면 됩니다. LazyCodex 생성 태그(#3425)가 있어도 로그 숫자·타임라인·설정 요약은 사람이 쓸 만한 수준입니다. types/config 분할로 닫을 이슈가 아닙니다.
우선순위 70은 풀 사용자에게 바로 체감되는 회귀/잔존 구멍이고, 수동 pause 워크어라운드만으로 버티게 두면 안 된다는 뜻입니다. 다만 업스트림 502 본문이 없어 “#3176 classifier miss”인지 “선택기가 stale active/affinity를 붙잡음”인지 한 번에 단정하진 마세요.
경로 src/codex/routing.ts
applyQuotaAutoSwitch/hasCodexQuotaHeadroom- 임계 로직은 HEAD에 있습니다. 보고는 이 경로가 새 admission에 안 먹거나, 먹어도 affinity가 되돌리는지 확인이 필요합니다.
경로 src/codex/quota.tsisCodexQuotaExhausted- short 100이면 소진입니다. 대시보드 숫자와 선택기 입력이 같은 스냅샷인지 봐야 합니다.
경로 src/server/relay.tsstreamAborted/ synthetic 502 - 이슈의 55건과 맞을 수 있습니다. same-request recovery 없음(recoveryKinds빈 배열)과 다음 요청 계정 선택이 따로 깨졌을 수 있습니다.
경로 #3029 / #3110 / #3176 / #585 - 관련 선례로 맞습니다. 중복 close보다는 “어느 가드가 2.41.0 경로에서 빠졌는지” 이슈로 유지하세요.메인테이너의 판단이 필요한 지점
- 2.42.0
dev에서 동일 시나리오를 재현해 이미 고쳐졌는지, 아니면 2.41.0과 같은지 - 1차 가설을 “선택기/affinity가 100% 계정을 계속 고름”으로 둘지, “본문 없는 502 recovery 공백”으로 둘지
- stickyRequestLimit=1과 데스크톱 affinity 키가 충돌하는지
- 핫픽스 범위를 pause-exhausted 자동 호출로 좁힐지, admission 제외로 넓힐지
너의 추천
open bug로 두고, 이슈에 적힌 회귀 네 줄을 테스트로 먼저 고정하세요. 임시 완화는 사용자가 한 것처럼 소진 계정 수동 pause(또는ocx account pause-exhausted)입니다. 본문 없는 502만으로 스냅샷 100%를 무시하지 않게 하는 쪽이 안전한 수정 방향입니다. 지금 close하지 마세요.이 댓글은 grok-bot이 작성했습니다
- 2.42.0
Investigated during the 260904 backlog review. I want to record what the code actually says, because it rules out the obvious fix.
It is not the quota selector.
tests/codex-routing.test.ts:325already pins "known 100% weekly usage is exhausted, not unknown, and switches accounts", and:308pinsisCodexQuotaExhaustedon an explicit 100% window. A known 100% account already rotates away today. The unknown-usage passes atsrc/codex/routing.ts:1215and:1655are deliberate — the comment above them says unknown usage must not drain a tier that was simply never primed — so tightening them would not fix your case and would break accounts whose providers report no quota at all.The suspect is that your failures were never recorded.
recordCodexUpstreamOutcomebegins with, atsrc/codex/routing.ts:2195:if (writerGeneration < lastReconciledGeneration && !liveHealthAccountIds.has(accountId)) return;
If a writer's captured generation is older than the last config reconcile and the account is not in the live set, the outcome is dropped whole — no
consecutiveFailuresincrement, nolastFailureAt, no soft-avoid. The transient branch at:2459never runs, soupstreamFailoverThreshold(default 3) is never reached regardless of how many 502s arrive. That fits your report precisely: 118 failures,sendCountall 1,recoveryKindsempty, and rotation only after a manual pause, which takes a different path.There is a second, independent amplifier at
:2455. The streak resets to 1 when the previous failure is older thanCODEX_FAILURE_WINDOW_MS(5 minutes). If your failing turns were more than five minutes apart — which hand-retried interactive work often is — the threshold of 3 is unreachable by construction, even when recording works perfectly.What would tell these apart, and it is the one thing I cannot get from the log summary you posted: did anything reload configuration between the accounts being registered and the failures arriving — a dashboard edit, an
ocx config set, an account add or pause, a service restart? A config reload bumps the generation, which is what arms the first path.If you still have the run-request records, the useful fields are the timestamps of consecutive failures (to see whether they fall inside a 5-minute window) and whether any config write happened in that period. Either answer produces a different fix, and I would rather fix the one that actually fired than harden both and hope.
- added a commit that references this issue
on Sep 4, 2026 Re-examined this against current
dev, and the earlier "quota selector is at fault" theory is still wrong — but so is the follow-up suspicion, which is worth recording so nobody re-walks it.Ruled out again: the quota selector.
tests/codex-routing.test.ts:325already proves a known-100% weekly account rotates away, andcomputeCodexUsageScore(src/codex/routing.ts:363-391) collapses short into long withMath.maxwhen a known long window exists.Newly ruled out: the generation guard.
src/codex/routing.ts:2195is an AND:if (writerGeneration < lastReconciledGeneration && !liveHealthAccountIds.has(accountId)) return;
liveHealthAccountIdsis filled at:331fromcontext.codexAccountIds, whichlistLiveCodexAccountIds(:282) builds from every id inconfig.codexAccounts. Account A was registered and live throughout, so the second condition is false and the early return never fires. This guard only reaches an account that was removed from config.Also ruled out: the failure-window amplifier.
:2455treats a failure as stale pastCODEX_FAILURE_WINDOW_MS(5 minutes,:116). Your timeline is 118 failures between 10:58:32 and 11:21:11 — roughly 12 seconds apart, so nothing ages out.So the two questions worth your time are different from the ones asked earlier:
- Was account A ever deleted and re-added during the incident, as opposed to the config merely being reloaded? Only removal empties it out of
liveHealthAccountIds. - Is the Codex App pinning a specific account on these requests?
src/codex/auth-context.ts:692,709setsmeta.fixedAccount = truewheneverfixedAccountIdis present, androuting.ts:2482/2487/2492then skips both affinity clearing and failover promotion. If that flag is on, 118 attempts on one account is the system behaving as designed, and the fix is a product decision about what "fixed" should mean when the fixed account is failing — not a routing bug.
Separately, one real defect surfaced while checking this, and it is not yours:
tests/codex-routing.test.ts:527never callsreconcileCodexRoutingHealth, so it asserts against an emptyliveHealthAccountIds— a state production never reaches. It pins an unreachable path green. It is not being changed in this pass because the fix belongs with whatever root cause this issue lands on, but it is recorded here so the next reader does not mistake it for coverage.- Was account A ever deleted and re-added during the incident, as opposed to the config merely being reloaded? Only removal empties it out of
Closing the demonstrated exhausted-account selection defect as implemented on
devin #3605 (commite1b9ec851958c46ad6210a989b62c7b367edefee).A freshly observed 100% short window now excludes the account even when the quota response omits its reset timestamp. Dedicated short-window observation time survives persistence and unrelated quota updates; stale, missing, or future observations remain unknown. The landed regression named “a 502 storm does not pin the pool to an exhausted account (#3425)” checks both new requests and an already-bound thread against a healthy alternate account.
This closes the reproduced selection mechanism, not a claim that every historical 502 had the same cause. Unmarked 502s remain transient and no blanket stream replay was introduced. If the original symptom recurs on a build containing this fix, reopen with the running revision and a sanitized current quota snapshot.
Verified that the landing commit is an ancestor of current
devaf50c6d3451078a7d298b044c08fd2684c9e8eeb. Cross-platform CI passed for that exact dev commit (Linux and macOS test shards; the Windows full-suite job was skipped).
Client or integration
Codex App
Area
Authentication and account pool
Summary
OpenCodex 2.41.0 continued routing new Codex requests to a ChatGPT pool account after its five-hour window reached 100%, although auto-switch was enabled at an 80% threshold and a second account had only 3% usage. Requests recovered immediately after the exhausted account was manually paused.
This appears to be a recurrence or remaining transport-path gap related to #3029, #3110, and #3176, rather than a missing auto-switch setting.
Reproduction
@bitkyc08/opencodex2.41.0 with two valid ChatGPT accounts in the Codex account pool.quotastrategy, set the threshold to 80%, and set the sticky request limit to 1.sendCount: 1and an emptyrecoveryKindsarray.Observed timeline on 2026-09-04 KST: account A produced 118 HTTP 502 attempts from 10:58:32 through 11:21:11. Account B returned HTTP 200 at 11:21:33, immediately after account A was manually paused.
Version
2.41.0
Operating system
Windows NT 10.0.26200.0
Provider and model
OpenAI ChatGPT account pool / Codex models used by Codex App
Logs or error output
The subsequent quota refresh showed account A at five-hour/short 100% and weekly 49%, while account B was at five-hour/short 3% and weekly 18%.
The exact upstream 502 response body was not persisted. Therefore, this report does not claim that every unmarked 502 contained the canonical quota message.
Screenshots and supporting files
No screenshot is attached. The values above were aggregated directly from the local OpenCodex usage log with account identifiers, prompts, tokens, and response content excluded.
Relevant prior work: #3029 reported the same manual-pause recovery symptom; #3110 treated a live full burst window as exhausted; #3176 added rotation for quota messages wrapped in HTTP 502; #585 added same-request failover for pre-stream 429/402 responses.
The 2.41.0
mainsource at4600f80198c8eb6cef1b9cf1338d51906100007econtains the bounded pre-stream 5xx quota-message classifier. It also explicitly leaves a quota failure terminal after SSE begins and records post-200 stream failure as synthetic HTTP 502 withstreamAborted.Suggested regression coverage:
shortPercent: 100, account B is healthy, quota threshold is 80%, and multiple new/concurrent requests are admitted.streamAbortedrequest may remain terminal, but subsequent new admissions must avoid account A.Redacted configuration
{ "accountPool": { "accounts": 2, "autoSwitch": true, "strategy": "quota", "thresholdPercent": 80, "stickyRequestLimit": 1 } }The JSON above is a redacted description of the effective dashboard settings, not a verbatim copy of the on-disk configuration schema.
Checks
This issue or PR was generated by LazyCodex.
Tag: lazycodex-generated