Skip to content

fix(codex): improve app-server exit diagnostics - #841

Merged
chuks-qua merged 2 commits into
mainfrom
fix/codex-exit-diagnostics
Jul 6, 2026
Merged

chuks-qua merged 2 commits into
mainfrom
fix/codex-exit-diagnostics

Conversation

@chuks-qua

@chuks-qua chuks-qua commented Jul 6, 2026 •

Copy link
Copy Markdown
Contributor

What

Improves Codex app-server crash diagnostics and Codex warning notification logging.

Why

When the Codex app-server exited unexpectedly, Mcode showed a raw exit code and dropped the child process stderr. Codex warning notifications also fell through the unknown-notification path, so their payloads were not recorded at warn level.

Key Changes

  • Retain a bounded, non-benign stderr tail for Codex app-server diagnostics.
  • Decode raw exit codes into signed and hex forms while keeping the existing exited unexpectedly banner trigger.
  • Include the latest stderr line in fatal messages and the bounded stderr tail in structured error logs.
  • Log Codex warning notifications at warn level with their payload.
  • Add regression coverage for stderr filtering, line and byte caps, oversized Unicode lines, graceful kills, warning notifications, and protocol coverage.

Verification

  • cd apps/server && bun run test src/providers/codex/__tests__/codex-app-server-handshake.test.ts src/providers/codex/__tests__/codex-event-mapper.test.ts src/providers/codex/__tests__/codex-protocol-coverage.test.ts: 3 files, 92 tests passed.
  • bun run verify: typecheck, lint, and unit tests passed.
  • Live stub check: a temporary Codex CLI stub wrote live stub stderr cause from issue 840 to stderr; CodexAppServer emitted an exited unexpectedly fatal with that line in latest stderr and stderrTail.

Fixes #840


View with Codesmith Autofix with Codesmith
Need help on this PR? Tag /codesmith with what you need. Autofix is disabled.

Summary by CodeRabbit

  • New Features

    • Added support for Codex warning notifications, which are now recognized and logged without creating downstream events.
    • Improved server crash and handshake diagnostics with richer exit details and recent stderr context.
  • Bug Fixes

    • Better handling of unexpected child exits, including clearer fatal messages.
    • Captured stderr history is now bounded more reliably, preventing overly large or broken log tails.

@coderabbitai

coderabbitai Bot commented Jul 6, 2026 •

Copy link
Copy Markdown

Review Change Stack

Warning

Review limit reached

@chuks-qua, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 16 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 3a330a6c-bdc8-4262-ba74-907f1e571fb0

📥 Commits

Reviewing files that changed from the base of the PR and between 8f2b7a3 and bdad990.

📒 Files selected for processing (3)
  • apps/server/src/providers/codex/__tests__/codex-app-server-handshake.test.ts
  • apps/server/src/providers/codex/codex-app-server.ts
  • apps/server/src/providers/codex/codex-types.ts
📝 Walkthrough

Walkthrough

CodexAppServer replaces its single last-stderr string with a bounded StderrTail buffer capturing multiple non-benign lines, and adds decoded exit diagnostics for unexpected child exits. CodexEventMapper adds a new WarningNotificationPayload type and logs "warning" notifications instead of treating them as unrecognized. Tests extended accordingly.

Changes

Codex exit diagnostics and warning notification handling

Layer / File(s) Summary
StderrTail buffer and exit diagnostics helpers
apps/server/src/providers/codex/codex-app-server.ts
Adds ExitDiagnostics type and StderrTail class bounding captured non-benign stderr lines by count and bytes, plus decodeExitDiagnostics and formatUnexpectedExitMessage helpers.
CodexAppServer wiring for stderrTail and exit reporting
apps/server/src/providers/codex/codex-app-server.ts
Replaces lastStderr field with stderrTail; updates handshake error enrichment, wireStderr() classification, and wireExit() to log decoded exit diagnostics and stderr tail, emitting an enriched fatal message.
Handshake/exit test harness and coverage
apps/server/src/providers/codex/__tests__/codex-app-server-handshake.test.ts
Extends fake child harness with a stderr stream and adds tests for unexpected-exit fatal messages, structured stderrTail log fields, bounded eviction by line/byte limits, surrogate-pair safety, and silent kill-requested teardown.
Warning notification type and mapper handling
apps/server/src/providers/codex/codex-types.ts, apps/server/src/providers/codex/codex-event-mapper.ts
Adds WarningNotificationPayload type extending CodexNotification, and an early branch in mapNotification that logs warning notifications and returns no events.
Warning notification test coverage
apps/server/src/providers/codex/__tests__/codex-event-mapper.test.ts, apps/server/src/providers/codex/__tests__/codex-protocol-coverage.test.ts
Adds a mapper test for warning notifications and updates protocol coverage's known methods/synthetic fixtures to include "warning".

Estimated code review effort: 3 (Moderate) | ~25 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Child as Codex child process
  participant AppServer as CodexAppServer
  participant StderrTail as StderrTail buffer
  participant Logger as logger

  Child->>AppServer: stderr line
  AppServer->>AppServer: classify (benign/non-benign)
  AppServer->>StderrTail: append non-benign line
  Child->>AppServer: exit(code, signal)
  AppServer->>AppServer: decodeExitDiagnostics(code, signal)
  AppServer->>StderrTail: latest()/snapshot
  AppServer->>AppServer: formatUnexpectedExitMessage(diagnostics, stderrTail)
  AppServer->>Logger: error(message, structured stderrTail fields)
  AppServer->>AppServer: emit "fatal" event
Loading

Possibly related PRs

  • Mzeey-Empire/mcode#247: Both PRs modify Codex provider core stderr/exit handling in codex-app-server.ts and notification mapping in codex-event-mapper.ts.
  • Mzeey-Empire/mcode#545: Both PRs touch CodexAppServer stderr capture/enrichment for handshake failures, replacing or adding last-stderr tracking mechanisms.
  • Mzeey-Empire/mcode#682: Both PRs adjust wireStderr/handshake-scoped fatal-stderr classification in the same file.

Poem

A rabbit sniffs the crashing trail,
No longer lost, no longer pale—
The last words stderr tried to say
Are caught and bounded, come what may.
Warnings whispered, now logged with care,
🐇 hop hop, the cause laid bare!

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title is concise and accurately summarizes the main change: improved Codex app-server exit diagnostics.
Description check ✅ Passed The description covers What, Why, Key Changes, and Verification; only the optional Config Changes section is omitted.
Linked Issues check ✅ Passed The changes match #840: exit codes are decoded, stderr is preserved and bounded, warning notifications are logged, and teardown stays silent.
Out of Scope Changes check ✅ Passed The PR stays focused on Codex diagnostics and tests; no unrelated functional changes stand out.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

The user-facing fatal message embedded the latest stderr line verbatim,
up to 16KB and attacker-influenceable. It flows into the web
CliErrorBanner, which lifts backtick-quoted substrings out as copyable
commands. Cap the excerpt at 256 code points and strip backticks; the
full bounded line stays in the structured stderrTail log fields.

Also remove dead threadId/turnId fields from WarningNotificationPayload
(warnings are logged wholesale, never routed per-thread), read the
latest stderr line through StderrTail.latest() in wireExit, and document
the single-line floor invariant in StderrTail.evict().
@chuks-qua
chuks-qua merged commit 13baf4f into main Jul 6, 2026
10 checks passed
@chuks-qua
chuks-qua deleted the fix/codex-exit-diagnostics branch July 6, 2026 12:18
@mze-bot mze-bot Bot mentioned this pull request Aug 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

PRD: name the cause when the Codex app-server exits unexpectedly

1 participant