fix(lifecycle): retryable restart-drain code and quiet journal no-op recovery

[lidge-jun]

Summary

Two lifecycle fixes that change what a client sees during recovery and restart.

Restart drain is retryable for every Codex version (closes #6642). While OpenCodex drains for a restart, new data-plane requests got HTTP 503 with error.code: "server_is_overloaded". Codex maps that code to ServerOverloaded ("Selected model is at capacity"). Older Codex clients never retry it, and current Codex retries it only while retry advice survives, so a restart of a few seconds could end the turn with a model-capacity message. The drain now answers:

{"error":{"type":"server_error","code":"server_restarting","message":"OpenCodex is restarting; retry this request."}}

It keeps HTTP 503, Content-Type: application/json, Retry-After: 5, and the receiving listener's CORS policy. Codex classifies the new code as a retryable UnexpectedStatus (openai/codex codex-rs/codex-api/src/api_bridge.rs, codex-rs/protocol/src/error.rs). Provider overload mapping in src/lib/errors.ts and the drain fence length (#6643) are unchanged.

Journal recovery stays quiet on a no-op and warns on an incomplete recovery. This carries the journal half of #6649 by @agentHits: restoreJournalState() reports whether it actually rewrote or removed a file, and reconcileJournal() no longer prints "did not shut down cleanly" when a stale journal needed no rewrite. On top of that, both reconcile branches (client owner and dead process) now print a content-free "recovery was incomplete; the journal was preserved" warning when restoration did not complete, for example when the generated profile cannot be removed. A quiet no-rewrite recovery also warns when the stale journal file itself could not be deleted (for example a Windows file lock), since removeJournal() ignores unlink errors. Before, such a partial recovery either claimed success or, with the carried change alone, went silent. The restart-admission half of #6649 is not carried; it stays on #6649 for maintainer review.

Co-authored-by: agentHits 140916359+agentHits@users.noreply.github.com

Verification

• bun test on issue-452-empty-503, codex-journal, codex-journal-recovery (new), shutdown-drain, system-restart, memory-watchdog, the test-layout pair, file-size-ratchet and structure-ssot: 204 pass / 0 fail on this head.
• Red/green: with the previous drainingResponse the updated drain test fails (0/1); with the previous src/codex/journal.ts all 11 new recovery tests fail, and the two journal-removal tests added in 359e100d1f fail on 552fa7f141; both pass on this head. The dead-owner tests stub process.kill(pid, 0) to throw ESRCH instead of relying on an unused PID, and profile-removal failure is injected with an EPERM unlinkSync mock.
• The drain test asserts allowed-origin and denied-origin CORS on the receiving listener, Retry-After: 5, JSON content type and the new code.
• bun run typecheck exit 0; bun run structure:check pass; bun run privacy:scan pass; docs-site build pass.
• src/server/index.ts shrinks from 887 to 885 lines (cap 893); scripts/test-layout/layout.json stays at 1,999 lines.
• Independent adversarial reviews (gpt-6.1-sol) passed both changes after folding their findings.
• The full local suite was not run because parallel sweep lanes share this machine; the remaining coverage is left to hosted CI on this head.

Checklist

• Scope stays focused and avoids unrelated cleanup.
• Docs or release notes were updated when needed. (reference/management-api.md, reference/architecture.md, troubleshooting/windows-memory.md, structure/codex-home.md, structure/ops/service-and-sidecars.md, structure/runtime.md)
• Security-sensitive changes were reviewed for secrets, auth, and unsafe defaults. (The drain response is sent before auth and keeps the receiving listener's CORS policy; new warnings carry no identifiers.)

Summary by CodeRabbit

• Bug Fixes
  • Restarting servers now return a consistent HTTP 503 response with a retry hint and the receiving listener’s CORS policy.
  • Journal recovery reports and warnings now reflect whether recovery completed and changed or removed files. Incomplete recovery retains its journal.
• Documentation
  • Updated lifecycle, troubleshooting, and API guidance to describe restart-drain responses and journal recovery behavior.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-06T13:25:38.983987Z	552fa7f	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: Repository: lidge-jun/opencodex/.coderabbit.yaml
• Review profile: ASSERTIVE
• Plan: Advanced
• Run ID: 08df6194-db0a-44a3-9326-23e675f155ea

📥 Commits

Reviewing files that changed from the base of the PR and between 552fa7f and 359e100.

📒 Files selected for processing (3)

• src/codex/journal.ts
• structure/codex-home.md
• tests/codex-integration/codex-journal-recovery.test.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 6 remain after this review.

---

📝 Walkthrough

Walkthrough

The changes distinguish actual file writes from already-original state during journal recovery. They also return a retryable server_restarting response during restart drains and apply the receiving listener’s CORS policy.

Changes

Journal recovery reporting

Layer / File(s)	Summary
Track file writes and report recovery src/codex/journal.ts	Restore results indicate whether config and profile files were rewritten or removed. Reconciliation reports recovery only when restoration completes and at least one file changes.
Validate and document recovery outcomes tests/codex-integration/codex-journal-recovery.test.ts, scripts/test-layout/layout.json, tests/fixtures/test-layout-expected.json, structure/codex-home.md, structure/runtime.md	Integration tests cover already-original state, file writes and removals, and incomplete recovery. Documentation describes write indicators and recovery outcomes; test-layout files register the new test.

Restart drain responses

Layer / File(s)	Summary
Return and verify listener-specific drain responses src/server/index.ts, tests/codex-integration/issue-452-empty-503.test.ts, docs-site/src/content/docs/reference/architecture.md, docs-site/src/content/docs/reference/management-api.md, docs-site/src/content/docs/troubleshooting/windows-memory.md, structure/ops/service-and-sidecars.md, structure/runtime.md	During a drain, the server returns HTTP 503 JSON with server_restarting, a retry message, Retry-After: 5, and CORS from the receiving listener. Integration tests check public and loopback listeners. Documentation describes the response and distinguishes it from provider overload errors.

Priority: ➖ Normal

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Bug fix · Severity of issue fixed: Medium

Merge Risk: ⚪ Minimal · up to 359e1

The change makes restart-drain responses retryable and quiets stale journal recovery that needs no rewrite. No merge-blocking risk is evident in the supplied context.

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name	Status	Explanation	Resolution
Out of Scope Changes check	⚠️ Warning	The only supplied directly linked issue is #6642, which concerns the restart-drain response. The journal recovery changes in src/codex/journal.ts, the recovery tests in `tests/codex-integration/code…	Move the journal recovery changes, their tests, and journal documentation to a separate PR for #6649, or remove them from this PR. Keep the changes that implement #6642’s restart-drain response.
Docstring Coverage	⚠️ Warning	Docstring coverage is 30.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 10 functions across 4 files. (1 skipped: …	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (3 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	For #6642, the change summary reports that src/server/index.ts now returns HTTP 503 with JSON code server_restarting, the restart message, and Retry-After: 5, while applying the receiving listen…
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly names both main changes: retryable restart-drain responses and quiet journal no-op recovery.

Full details: Out of Scope Changes check

Explanation

The only supplied directly linked issue is #6642, which concerns the restart-drain response. The journal recovery changes in src/codex/journal.ts, the recovery tests in tests/codex-integration/codex-journal-recovery.test.ts, and journal documentation in structure/codex-home.md and structure/runtime.md do not implement or support that response. The PR description associates this work with #6649, but #6649 is not supplied as a directly linked target with coding requirements.

Full details: Docstring Coverage

Explanation

Docstring coverage is 30.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 10 functions across 4 files. (1 skipped: 1 unsupported.)

• Fix all pre-merge checks with AI

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 552fa7f141

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[github-actions]

✅ Deterministic PR hygiene checks passed.

[lidge-jun]

Maintainer integration into dev (MAINTAINERS.md dev-only maintainer integration; not a self-approval or an independent review).

• Exact head: 359e100d1fe7a4be71c16736a6a95d254643bb8f
• Hosted CI on that head: all 30 latest check runs completed and every applicable one succeeded. Skipped jobs are gated off by their own path/event conditions.
• fix(codex-shim): name the path and chmod fix when a private shim artifact is group-writable #6680, fix(service): credit Linux desktop Start at Login as restart protection in doctor #6679 and fix(lifecycle): retryable restart-drain code and quiet journal no-op recovery #6686 share structure/runtime.md, and fix(service): credit Linux desktop Start at Login as restart protection in doctor #6679/fix(lifecycle): retryable restart-drain code and quiet journal no-op recovery #6686 also share scripts/test-layout/layout.json, tests/fixtures/test-layout-expected.json and structure/ops/service-and-sidecars.md. The coordinator merged all three onto origin/dev in this order in a scratch worktree: clean auto-merge, layout.json at 1998 lines, and on that exact union tree bun test of the layout/tooling/file-size-ratchet guards (27 pass) and of the three PRs' changed test files (86 pass), bun run structure:check and bun x tsc --noEmit all passed.
• Review: independent gpt-6.1-sol review in the compat sweep lane (PASS); no unresolved review threads, no maintainer change requests.
• Scope: restart drain answers 503 with Retry-After and the retryable server_restarting code instead of server_is_overloaded (closes [Bug] A local restart drain answers with server_is_overloaded, so Codex reports model capacity instead of retrying #6642), plus the journal no-op recovery part of fix: update restart with stale service record; silent journal no-op recovery #6649 carried with Co-authored-by @agentHits; no gui/ paths.
• Full local suite not run (parallel lanes share one machine); the dev push CI after this merge is the remaining check.