Skip to content

Bug: close remaining stored Pool 401 concurrency and recovery-budget gaps #2892

Description

@luvs01

Client or integration

Codex App and Codex CLI through the OpenCodex Responses/compact routes.

Area

Authentication and account pool

Summary

#2889 fixed the primary reported failure: an ordinary stored Pool account now gets one generation-fenced forced refresh and one same-account replay after a pre-stream 401. It merged as 0f4cd2a0be46178be91689096cd0a00bfeb9337b.

Five bounded concurrency and logical-request-budget gaps remain in that merged path:

  1. A superseding stored generation can be returned without checking that it is fresh and not the rejected bearer.
  2. The initiating caller's abort signal is part of the grant-wide refresh flight, so cancelling one waiter can abort live joiners.
  3. A rotated refresh grant reaches the owner and active joiners but not an inactive same-account alias that still holds the old grant.
  4. Routing validates the credential generation and then mutates health, needsReauth, and affinity separately; another process can replace the credential between those operations.
  5. After the stored-account refresh/replay budget is consumed, a replay 429, 402, or other eligible failure can continue to another Pool account, remembered model, or combo target.

Expected behavior:

  • each waiter owns only its own cancellation while the shared refresh uses an internal lifecycle signal;
  • a superseding winner is independently fresh and not the rejected bearer;
  • rotated grants are CAS-merged into unchanged same-account aliases without overwriting a concurrently newer access credential;
  • generation validation and credential side effects are atomic or generation-scoped and discarded when stale;
  • the replay response is authoritative for the rest of that logical request, with no later account/model/combo send.

Implementation should stay split into two independent PRs based directly on current dev:

  1. pooled credential concurrency and generation safety;
  2. one recovery budget for a stored Pool 401.

Do not replace the merged #2889 state machine wholesale. Preserve its forced-refresh provenance, rejected-bearer exclusion, and affinity handoff behavior.

Reproduction

The gaps are deterministic at the code/interleaving level and were reverified against the exact merge tree:

  1. Configure Pool mode with a stored account whose time-valid access token receives a pre-stream 401.
  2. Start two refresh waiters sharing one grant, then cancel the initiating waiter. The current shared AbortSignal.any(...) can cancel the token request for the still-live waiter.
  3. During the flight, persist a newer but expired G+1 credential. The joined-flight supersession branch can return it without the normal freshness test.
  4. Rotate the refresh token while another same-account alias is inactive. Only the owner/live joiner record is updated, leaving the inactive alias on the invalidated grant.
  5. Between isCodexAccountGenerationLive(...) and the routing side effects, persist G+1 from another process. Stale G evidence can still affect in-memory health/reauth/affinity.
  6. Return 429 or 402 from the same-account replay. Compact proceeds to its alternate-account/remembered-model path; regular Responses can re-enter later recovery and combo ladders.

Required regressions:

  • owner cancellation does not cancel a live joiner;
  • stale/same-bearer G+1 is not replayed;
  • inactive alias grant rotation preserves a concurrently newer access credential;
  • a forced generation interleaving cannot retire G+1;
  • replay failure produces zero subsequent account/model/combo sends;
  • existing one-refresh/one-replay and next-request availability tests remain green.

Version

dev@0f4cd2a0be46178be91689096cd0a00bfeb9337b (merged #2889 tree)

Operating system

Cross-platform state-machine defect; reverified from Windows 11 against the shared TypeScript implementation.

Provider and model

OpenAI Codex login Pool; any model using native Responses or compact routing.

Logs or error output

No secrets or raw request logs are required. The failing paths are the merged source interleavings described above.

Screenshots and supporting files

No open issue or PR matched these exact remaining contracts when rechecked after #2889 merged.

Redacted configuration

{
  "codexAccountMode": "pool",
  "storedAccounts": "two or more aliases; no tokens or account identifiers included"
}

Checks

  • I searched existing issues and documentation.
  • I removed secrets, tokens, account details, request credentials, and personal data.

Activity

  1. github-actions commented on Aug 29, 2026

    @github-actions
    Contributor

    Issue reopened

    The report now contains the information required by the automated check. Thanks for updating it.

  2. lidge-jun commented on Aug 29, 2026

    @lidge-jun
    Owner

    리뷰 · 우선순위 72 / 80

    이 이슈는 #2889가 dev에 들어간 뒤에도 남은 구멍을 다섯 개로 나눈다. HEAD는 0f4cd2a0be46178be91689096cd0a00bfeb9337b이다. 그 커밋은 저장 풀 계정의 사전 스트림 401을 격리 대신 강제 refresh와 같은 계정 replay로 바꿨다. 제보된 #2887 구멍은 막혔다. 이 이슈가 말하는 것은 그 새 길이 아직 닫지 못한 동시성과, replay 이후 논리 요청 예산이다. 네이티브 메인(#2848)이나 Kiro 쿼터 기차와 섞지 말라고 한 경계는 지금 HEAD와도 맞다.

    앞의 네 개는 자격 증명 저장 쪽이다. 첫째, 합류한 refresh 비행이 더 새 generation을 돌려줄 때, 그 값이 신선한지와 방금 거절된 bearer가 아닌지를 다시 안 볼 수 있다. 둘째, 공유 refresh의 AbortSignal에 시작한 쪽의 취소가 들어가면, 아직 기다리는 다른 호출의 토큰 요청까지 같이 죽을 수 있다. 셋째, refresh grant가 돌아가도 같은 계정의 잠든 별칭이 옛 grant에 남을 수 있다. 넷째, src/codex/routing.ts가 generation을 확인한 뒤 health/needsReauth/affinity를 따로 바꿔서, 그 사이에 다른 프로세스가 G+1을 쓰면 낡은 증거가 새 자격을 건드릴 수 있다. 이 네 개는 이번 웨이크의 PR #2895가 건드리지 않는다. 본문도 두 독립 PR로 나누라고 적었다.

    다섯 번째는 복구 예산이다. HEAD src/server/responses/core.ts는 저장 replay가 429/402면 retryCodexPoolOnAlternateAccount로 다른 계정을 부른다. compact는 대체 계정과 기억된 모델 핸드오프가 있고, 콤보 comboFailureDecision과 정책 후보는 429를 hop 한다. 그래서 “같은 계정 한 번”이 그 요청의 끝이 아니다. 이 조각은 이미 PR #2895가 맡았다. 저장 replay가 나간 뒤 그 실패를 권위 있는 결과로 보고, 다른 계정·모델·콤보·정책으로 안 넘어가게 한다. 네이티브 메인의 더 넓은 복구는 유지한다. 이슈를 #2895로 닫으면 안 된다. #2895는 Refs #2892이지 Closes가 아니다.

    수락 목록도 지금 방향과 맞다. 시작한 쪽의 취소가 산 합류자를 죽이지 말 것, 낡은/같은 bearer G+1을 replay하지 말 것, 잠든 별칭 grant 회전이 더 새 access를 덮지 말 것, 끼어든 generation이 G+1을 퇴직시키지 말 것, replay 실패 뒤에 계정/모델/콤보 추가 전송이 0일 것, #2889가 넣은 1-refresh/1-replay와 다음 요청 선택 테스트는 그대로 초록일 것. #2889 상태 기계를 통째로 바꾸지 말라는 말도 맞다. 강제 refresh 출처, 거절된 bearer 제외, affinity G→G+1 핸드오프는 유지해야 한다.

    이 이슈는 types.ts/config.ts 분할과 무관하다. 닫을 중복도 아니다. #2887은 이미 #2889로 닫혔고, 여기는 그 후속이다. 라벨 bug / account-pool도 맞다.

    경로 #2895 - 복구 예산(이슈 항목 5)만 구현한다. 합쳐도 이 이슈를 닫지 마라.

    경로 src/codex/account-store.ts forceRefreshCodexPoolToken - 항목 1·2·3(신선한 승자, 공유 비행 취소, 별칭 grant 전파)의 자리. 아직 열린 PR이 없다.

    경로 src/codex/routing.ts 자격 부작용 - 항목 4. generation 확인과 health/needsReauth/affinity 쓰기가 한 펜스가 아니다.

    경로 src/server/responses/core.ts 약 4185줄·compact.ts 약 838줄·src/combos/failover.ts 약 174줄 - 항목 5의 HEAD 구멍. #2895가 여기를 잠근다.

    경로 #2889 사이드카 기록 펜스 - 이 이슈의 다섯 항목 밖이다. 사이드카 401 기록에 credentialGeneration이 없는 것은 알려진 잔여이고, 여기 수락 목록에 넣지 마라.

    메인테이너의 판단이 필요한 지점

    • #2895를 이 이슈와 따로 먼저 dev에 넣을지
    • 남은 동시성 네 조각을 한 PR로 받을지, grant 비행/별칭 전파/routing 원자성을 더 쪼갤지
    • 실패한 refresh(replay 미전송)를 복구 예산에 넣을지, 콤보 hop으로 둘지

    너의 추천
    이 이슈는 열어 두어라. 항목 5는 #2895를 호스트 테스트 초록 뒤 합치면 된다. 항목 1~4는 #2889 상태 기계를 갈아엎지 않는 별도 버그 PR로 받아라. 이 이슈를 #2887이나 메인/WHAM 이슈로 합치지 마라. types/config 분할 때문에 닫을 대상이 아니다.

    이 댓글은 grok-bot이 작성했습니다

  3. lidge-jun commented on Aug 29, 2026

    @lidge-jun
    Owner

    Gap 5 is done — landed on dev as 4fa981f (#2897, carrying @luvs01's #2895).

    After a stored Pool account's 401 is refreshed and replayed once, that replay's result is now
    authoritative for account, model, and combo selection: no later send can reach a different Pool
    account, a remembered compact model, a combo target, or a policy candidate.

    One correction to how the gap was originally framed here. The expected behaviour I wrote was "no
    later account/model/combo send", and the first implementation read that literally — one break on any
    stored-replay 4xx. That also stops two ladders that send to the account already paying: the
    allow-listed gated-model 400 retry against a still-entitled account, and the one-shot opaque-blob
    rebuild. Neither charges another account, so neither was ever in scope, and cutting them turned a
    recoverable invalid_encrypted_content into a user-visible 400. The bound that shipped is stated by
    account, not by further sends.

    Still open, as the separate concurrency PR this issue asks for:

    1. a superseding stored generation can be returned without proving it is fresh and not the rejected
      bearer;
    2. the initiating caller's abort signal is part of the grant-wide refresh flight, so one waiter's
      cancellation can abort live joiners;
    3. a rotated refresh grant reaches the owner and active joiners but not an inactive same-account
      alias still holding the old grant;
    4. routing validates the credential generation and then mutates health, needsReauth, and affinity
      separately, so another process can replace the credential in between.
  4. Ingwannu commented on Aug 29, 2026

    @Ingwannu
    Owner

    All five reported gaps are now on dev:

    #2934 was reviewed at exact head bd26bcd393d07f6a25f40b3a104c54a5f646cc82, passed the focused concurrency/routing evidence and all hosted checks including macOS, and was merged only after its credential-boundary rationale was corrected to match the shipped design.

    Closing as completed.

  5. lidge-jun commented on Aug 29, 2026

    @lidge-jun
    Owner

    Gaps 3 and 4 are on dev as f5b8529f73d9c1468ce3a005ecd26c4be12d1516 (#2934), reviewed and merged by a non-author maintainer. That completes the set alongside gaps 1-2 (8f199fcb6, #2920) and gap 5 (84049830e, #2922), so I am closing this.

    Gap 4 is closed. A 401/403 now carries the credential generation it was derived from, and that provenance is judged when the evidence is read rather than when it is written. credentialFailureGeneration lives on the health entry itself, reauthAccounts maps an account id to the generation that justified the mark, and a spent entry is discarded at getCodexUpstreamHealth, at shouldFailover, and at recordCodexUpstreamOutcome entry. A stale 401 can no longer quarantine the credential that replaced it, nor donate its failure count to it. recordCodexUpstreamOutcome is still synchronous and still never takes the config lock on the request path.

    Your async interleaving as literally written is not reachable — that function has no await between the check and the mutations — but the cross-process version is real without one, because the check is an unlocked read while another process holds the mutation lock. That is what got fixed.

    Gap 3 is a partial close, and I want to be exact about what remains. A refresh now propagates the rotated grant to same-account aliases in one lock acquisition and one persist, so no record is left holding a grant upstream has already invalidated. But an alias is repaired only when it is provably an untouched duplicate: same pre-refresh fingerprint, same access token, same expiry, and a non-empty chatgptAccountId equal to the owner's.

    A mixed alias — same refresh grant, but its access token has moved on — is still not repaired. I tried, and an audit rejected it for two reasons I could not answer:

    • A higher generation is how the code proves an access-token JWT is newer, which is what lets a JWT plan claim supersede an older WHAM observation. Bumping the generation while keeping the old access token would let a stale JWT overwrite an authoritative plan.
    • Refresh flights are keyed by grant and do not record their participants, so a scan cannot distinguish a dormant record from a live joiner. Rotating a joiner's grant while preserving its already-401-rejected bearer would hand that rejected token straight back on the retry — breaking 401 recovery in exactly the case it exists for.

    Doing that case properly needs durable grant lineage plus verified identity binding, which the current credential model cannot express safely. If you are seeing a retired account whose access token had diverged, that is the uncovered case and it is worth a separate issue with the account state at the time — please do open one rather than reopening this.

    One thing I could not verify from repository source: whether OpenAI's token service can bind a single refresh grant to two different account ids. The code compares identity rather than assuming it cannot, and absent or empty identity fails closed.

    Thanks for the precise report — the reproduction detail is what made the narrow-eligibility argument decidable instead of guesswork.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    account-poolOAuth, credentials, Codex pool, quota, failover, plansbugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions