[Feature]: opt-in reset-credit recovery after verified pre-stream quota exhaustion #657

Area

Authentication / account pools / quota recovery / request replay

Goal

Allow OpenCodex to recover an unexposed Codex turn after the upstream has positively identified quota exhaustion by optionally consuming one real reset credit and replaying that same logical request.

The trigger is the authoritative upstream rejection.

Displayed quota percentages, local usage estimates, generic rate limits, transient server errors, and authentication failures must not authorize reset-credit consumption.

Automatic redemption must remain opt-in and disabled by default because a reset credit is scarce and irreversible.

Current status

This feature is partially implemented. Two safety foundations are now merged into dev, but the end-to-end automatic redemption and replay path is not wired yet.

Delivered: verified reset-eligible exhaustion classification

PR #866 merged as a088e4b1:

feat(codex): classify reset-eligible quota rejection

OpenCodex can distinguish verified reset-eligible pre-stream exhaustion from unrelated failures.

Only exact structured exhaustion signals such as:

usage_limit_exceeded
insufficient_quota

on an eligible pre-stream 429 / 402 response may be classified as:

reset-eligible-exhaustion

The classifier fails closed for malformed, truncated, oversized, cancelled, ambiguous, or untrusted error bodies.

Generic 429, unverified 402, transient 5xx, 401, 403, and other upstream failures do not become reset-credit eligible merely from status or message text.

This classification boundary must be reused rather than reimplemented.

Delivered: reset-credit recovery safety coordinator

PR #1410 merged as d91ca60c72f70f08c479da174910581657ed783c:

feat(codex): coordinate reset-credit recovery attempts

This adds the coordinator-only safety foundation for the scarce, irreversible reset-credit operation. It proves and enforces the core invariants before runtime wiring:

• one recovery attempt per logical turn;
• one stable UUIDv4 operation identity reused across bounded consume retries;
• process-shared single-flight coordination for the same account and exhaustion generation;
• generation-bound recovery with revalidation before irreversible dispatch;
• fail-closed cancellation before dispatch;
• output-exposure guards that prevent recovery after client-visible output has committed;
• deterministic handling of reset, already_redeemed, nothing_to_reset, and no_credit;
• bounded retries, deadlines, waiters, process state, and orphaned adapter execution accounting;
• monotonic terminal fences so uncertain irreversible outcomes cannot blindly spend another credit;
• fail-closed handling for malformed adapters, signals, stale generations, contract mismatches, and capacity saturation.

PR #1410 is intentionally not the runtime integration. It does not automatically consume a credit, replay a Responses request, add recovery policy/config, select alternate accounts, or expose UI controls.

Remaining implementation

The next slices should wire the proven coordinator into the real request path without weakening its invariants.

At minimum, the remaining work is:

• add explicit opt-in policy, defaulting off;
• define recovery priority such as alternate-first and/or reset-first;
• invoke the real reset-credit consume path only after verified pre-stream reset-eligible exhaustion;
• refresh authoritative account/quota state after a successful or idempotent-compatible redemption;
• replay the same complete logical request at most once;
• preserve trustworthy replay state, including previous_response_id continuations;
• reuse the generic account-pool routing/failover contract from [Feature]: Generic OAuth account-pool failover with session affinity and quota-aware selection #695 for alternate-account selection;
• prevent detached replay after client cancellation, disconnect, or proxy shutdown;
• add bounded, sanitized recovery diagnostics;
• expose config/UI management controls only after runtime integration is proven safe.

Existing alternate-account recovery

OpenCodex already has bounded pre-stream alternate-account retry behavior from #584 / #585.

That behavior remains separate from spending a reset credit.

A generic 429 or 402 may still participate in the existing alternate-account retry policy where currently supported, but it must not authorize an irreversible reset-credit operation.

This issue owns the additional recovery step that may occur only after:

verified reset-eligible exhaustion

has been established.

Desired policy

Add an explicit opt-in policy for reset-credit recovery.

Conceptually:

{
  "codexQuotaRecovery": {
    "enabled": true,
    "autoRedeemResetCredit": true,
    "priority": "alternate-first"
  }
}

The exact configuration names may follow existing conventions.

At minimum:

enabled = false

must remain the default.

If automatic redemption is enabled, support an explicit recovery priority such as:

alternate-first
reset-first

Alternate first

account A
  -> verified reset-eligible exhaustion
  -> try eligible alternate account
  -> no alternate succeeds
  -> redeem at most one reset credit for A
  -> replay A once

Reset first

account A
  -> verified reset-eligible exhaustion
  -> redeem at most one reset credit for A
  -> replay A once
  -> if recovery does not succeed, try eligible alternate

The exact candidate-selection mechanics should reuse the account-pool routing contract rather than create an independent failover engine.

Hard safety boundary: pre-stream only

Automatic redemption and replay are allowed only before any model-visible effect has escaped.

Do not auto-replay after OpenCodex has exposed any assistant output, reasoning output, tool call, tool result, Responses output item, terminal event, or other client-visible model event.

Once output is committed, automatic replay could duplicate text or tool side effects.

The recovery contract is:

request admitted
      ↓
upstream rejects before output
      ↓
verified reset-eligible exhaustion
      ↓
recovery may run

not:

partial response emitted
      ↓
failure
      ↓
replay

Replay completeness

The same logical request may be replayed only when OpenCodex still has a complete, trustworthy replay source.

This includes requests whose context depends on:

previous_response_id

If the required continuation/history cannot be expanded or reconstructed safely, fail closed rather than replaying an incomplete conversation.

Recovery must preserve request body semantics, conversation context, model selection, relevant account affinity semantics, tool definitions, and protocol metadata required by the original request.

Do not reconstruct a “similar” request from partial diagnostics.

Redemption outcomes

Use the existing reset-credit mechanism and its normalized outcomes:

reset
already_redeemed
nothing_to_reset
no_credit

reset

Refresh authoritative account/quota state and allow one bounded replay.

already_redeemed

Treat this as idempotent-compatible only when it belongs to the same logical recovery generation. Refresh state before replay.

nothing_to_reset

Do not fabricate success. Continue to the next permitted recovery step or return the original failure.

no_credit

Do not repeatedly retry redemption. Continue to an eligible alternate if policy allows, otherwise return the failure.

Safety invariants

The coordinator from #1410 now provides the foundation for these invariants, and runtime wiring must preserve them:

• at most one reset credit automatically consumed per logical turn;
• one idempotency identity reused across retries of the same redemption operation;
• concurrent requests for the same account/exhaustion generation cannot independently consume several credits;
• cancellation before redemption dispatch prevents redemption and replay;
• an uncertain consume result cannot authorize blind second-credit consumption;
• account/quota state is revalidated immediately before irreversible redemption;
• recovery is bounded and cannot recurse indefinitely;
• no replay may occur after output exposure.

Cancellation, disconnect, and shutdown

If the client disconnects or cancels before redemption is dispatched:

do not redeem
do not replay

If redemption has already been sent upstream, OpenCodex must report/record its outcome truthfully and must not assume it can undo the credit consumption.

A cancelled request must never leave behind a background replay that later sends a model request with no attached client.

Proxy shutdown must obey the same rule.

Failure classes that must stay separate

Generic rate limiting

429, Retry-After, or rate_limit_error without verified reset eligibility use the existing rate-limit/failover behavior. Do not spend a reset credit.

Transient provider failure

500, 502, 503, 504, 520, 521, 522, overload, and temporary availability failures use the existing transient retry/failover policy. Do not spend a reset credit.

Authentication / permission

401 / 403 use the appropriate reauthentication or permission behavior. Do not spend a reset credit.

Connection failures

Connection resets, stale sockets, DNS errors, and similar transport failures are not quota evidence. Do not spend a reset credit.

Relationship to generic account failover

#695 owns the generic OAuth account-selection, health, affinity, and failover engine.

This issue should consume that engine for alternate-account selection. It must not introduce a second competing account-pool state machine merely for reset-credit recovery.

Relationship to other quota automation

#822

Tracks proactive time-based redemption before a reset credit expires. Its trigger is credit expiry, not verified upstream quota rejection.

#823

Tracks post-reset activation intended to establish the next quota window. That happens after a natural reset and is separate from failed-turn recovery.

Natural-reset waiting

Waiting minutes for an upcoming natural quota reset while holding the original client request open has a materially different lifecycle contract and remains out of scope here.

Observability

Record bounded, sanitized recovery diagnostics such as:

trigger: reset-eligible-exhaustion
priority: alternate-first
alternateAttempted: true
redemptionAttempted: true
redemptionOutcome: reset
replayAttempt: 1
finalOutcome: success

Do not log email addresses, OAuth tokens, refresh tokens, raw account IDs, raw upstream response bodies, or reset-credit authorization material.

Use safe account references where correlation is required.

Management surfaces

After runtime integration is proven, expose the policy consistently through supported management surfaces.

Users should be able to determine:

• whether automatic reset-credit recovery is enabled;
• which recovery priority is selected;
• that at most one credit may be consumed per unexposed logical turn.

The UI must make clear that enabling the feature permits OpenCodex to consume a scarce reset credit automatically.

Do not bundle runtime safety work, account recovery engine, Dashboard controls, and documentation into one implementation PR.

Out of scope

This issue does not own:

• semantic rejection classification already delivered by feat(codex): classify reset-eligible quota rejection #866;
• generic OAuth account failover policy from [Feature]: Generic OAuth account-pool failover with session affinity and quota-aware selection #695;
• generic rate-limit retry;
• transient 5xx retry;
• natural-reset waiting;
• credit-expiry redemption from [Feature]: opt-in crash-safe reset-credit auto-redemption before expiry #822;
• post-reset account activation from [Research] Provider-specific post-reset activation to anchor quota windows #823;
• proactive percentage-driven account switching;
• recovery after client-visible output has begun.

Acceptance criteria

Delivered foundation

• Reset-eligible pre-stream exhaustion has an explicit semantic classification.
• usage_limit_exceeded / insufficient_quota are distinguished from generic rate limiting.
• Generic 429 and unverified 402 do not authorize reset-credit consumption.
• 5xx, auth, permission, malformed, and ambiguous responses fail closed for reset-credit eligibility.
• Classification is bounded and does not consume the original upstream response.
• One recovery attempt is reserved per logical turn.
• One stable operation identity is reused across bounded consume retries.
• Concurrent same-account/same-generation attempts are single-flighted.
• Cancellation before coordinator dispatch fails closed.
• Output exposure prevents recovery dispatch.
• Account/exhaustion generation is revalidated before irreversible dispatch.
• reset, already_redeemed, nothing_to_reset, and no_credit have deterministic coordinator outcomes.
• Coordinator state, retries, waiters, deadlines, and retained executions are bounded.
• Focused coordinator concurrency, cancellation, idempotency, generation, failure, and re-entrancy regressions exist.

Remaining runtime integration

• Automatic reset-credit recovery is explicit opt-in and defaults off.
• alternate-first and/or reset-first policy is defined explicitly.
• The coordinator is wired into the real pre-stream Responses/Codex failure path.
• Real redemption occurs only for verified reset-eligible pre-stream exhaustion.
• No replay occurs after client-visible output has begun.
• Replay requires complete trustworthy conversation/request state.
• previous_response_id replay is expanded/reconstructed safely or fails closed.
• Successful/idempotent-compatible redemption refreshes authoritative account/quota state before replay.
• The same logical request is replayed at most once.
• Client disconnect cannot leave an orphan background replay.
• Proxy shutdown cannot leave a later detached replay.
• Alternate-account selection reuses the generic account-pool policy.
• Runtime recovery attempts are bounded and cannot recurse indefinitely.
• Diagnostics expose recovery state without credential/account leakage.
• End-to-end integration tests cover redemption, replay, disconnect/shutdown, alternate policy, and failure classes.
• Management controls are added only after runtime integration is proven.
• Accepted end-to-end implementation is merged into dev.

Related

• feat(codex): classify reset-eligible quota rejection #866 — merged semantic reset-eligibility classifier.
• feat(codex): coordinate reset-credit recovery attempts #1410 — merged reset-credit recovery safety coordinator.
• Multi-account failover issue: Session freezes on depleted primary account despite active secondary Pro account #584 / fix(codex): same-request multi-account failover on quota 429 (#584) #585 — existing pre-stream alternate-account retry foundation.
• [Feature]: Generic OAuth account-pool failover with session affinity and quota-aware selection #695 — generic OAuth account-pool selection/failover.
• [Feature]: opt-in crash-safe reset-credit auto-redemption before expiry #822 — proactive reset-credit redemption before expiry.
• [Research] Provider-specific post-reset activation to anchor quota windows #823 — post-reset activation after natural quota reset.