Codex diff snapshots can accumulate in unbounded ingestion queues #12883

Activity

1. juliusmarminge commented on Sep 21, 2026

   

   juliusmarminge

   on Sep 21, 2026

   Member

   More actions

   Triage

   Confirmed on current main (1de563c14, after the reported nightly 7445aa733ada). This is a real unused-payload retention path. It is not fixed by #12305, and it is not a duplicate of #12758 or #12884.

   #12305 only bounds diagnostic provider log records. Native turn/diff/updated frames are now skipped entirely (transientNativeMethods). Canonical turn.diff.updated is still persisted, but #12305 clamps the serialized line. The runtime objects that feed the queues are unchanged.

   What the code does

   1. CodexAdapter maps a native turn/diff/updated onto a canonical event that keeps the same string twice:
      • raw.payload.diff via runtimeEventBase
      • payload.unifiedDiff in the turn/diff/updated mapper
        (CodexAdapter.ts ~975–993, ~1664–1676)
   2. That full event then sits in these unbounded buffers:
      • CodexSessionRuntime: Queue.unbounded<ProviderEvent>()
      • CodexAdapter: Queue.unbounded<ProviderRuntimeEvent>()
      • ProviderService: PubSub.unbounded<ProviderRuntimeEvent>()
      • ProviderRuntimeIngestion diffWorker and, after repo detection, the lifecycle worker (makeDrainableWorker → TxQueue.unbounded)
   3. recordProviderDiff never reads unifiedDiff or raw.payload.diff. It dispatches a placeholder checkpoint: status: "missing", files: []. After the first placeholder for a turn, later snapshots are dropped (hasCheckpointForTurn).
   4. fix(server): keep VCS waits from blocking turn completion #11970 moved checkpointStore.isGitRepository onto the dedicated diffWorker so a hung git no longer stalls turn completion. That also means the worker can accept snapshots faster than it can drain them, and each queued item still holds the full strings.
   5. Clients never read unifiedDiff. The field exists only on the contracts schema (TurnDiffUpdatedPayload) and in server fixtures/tests.

   CheckpointReactor also subscribes to ProviderService.streamEvents but immediately drops turn.diff.updated. PubSub still delivers the full object to that subscriber.

   Only Codex emits this event type.

   What this is not

   The report is careful, and the code agrees: this is a reachable unbounded path, not a proven crash cause. Provider logs do not record queue depth. The ~49 million-character diffs cited here are the durable Codex file-change payloads from #12758, which shows this provider can emit huge diff strings on the affected install. They are not measured turn/diff/updated queue entries.

   Related, not duplicates:

   • Oversized Codex diff event exhausts desktop backend heap #10924 / fix(server): bound provider event log records before serialization #12305 — logger bound only. The Oversized Codex diff event exhausts desktop backend heap #10924 triage already noted orchestration does not store unifiedDiff and asked to stop the dual-field copy. fix(server): bound provider event log records before serialization #12305 did not do that.
   • fix(server): stop an oversized provider diff from exhausting the backend heap #10965 / fix(server): stop an oversized provider diff from exhausting the backend heap #10968 / fix(server): stop an oversized provider diff from exhausting the backend heap #11125 (closed) — would have replaced raw.payload.diff with a marker and still kept the full unifiedDiff on the runtime event.
   • fix(server): keep VCS waits from blocking turn completion #11970 — isolated the VCS wait; that is the accumulate-while-git-is-slow path.
   • Codex file-change diffs are duplicated in SQLite and exhaust backend heap #12758 — durable SQLite copies of item.started / item.completed file-change changes[].diff.
   • Codex app-server input has no per-message byte limit #12884 — missing app-server per-message byte limit before parse.
   • Background service exhausts V8 heap and aborts after ~1-2 days of continuous uptime under sustained session load #12584 / [Bug]: Desktop backend still SIGABRTs with V8 OOM after 9–18h on 0.0.35 (hydration bounds already present) #8648 — broader long-uptime / large-profile heap exhaustion.
   • [Bug]: app-server stdin reader is quadratic in line length — OOM crash on large tool payloads #5389 — quadratic app-server reader. Fixed.
   • Checkpoint diffs never resolve: ProviderRuntimeIngestion placeholder preempts CheckpointReactor git capture #585 — placeholder checkpoint ordering, not memory bounds.

   Next step

   Accept. Slim the event to the fields recordProviderDiff actually uses (event / thread / turn identity and time) before the first queue, or at least before diffWorker.enqueue. Do not keep raw.payload.diff or payload.unifiedDiff on queued work. Coalesce repeated snapshots for the same thread and turn (KeyedCoalescingWorker already exists). A later snapshot after the first placeholder is already a no-op.

   Add a regression that a generated large params.diff never remains on queued ingestion work, and that many updates for one thread+turn stay bounded.

   Restarting the backend clears the live queues. It does not close the path.

   Not closing.
2. juliusmarminge

   added

   acceptedfeature request acceptedfeature request accepted

   bugSomething is broken or behaving incorrectly.Something is broken or behaving incorrectly.

   via-triageFiled through npx t3 triageFiled through npx t3 triage

   on Sep 21, 2026
3. t3dotgg commented on Oct 2, 2026

   

   t3dotgg

   on Oct 2, 2026

   Member

   More actions

   Note

   🤖 GPT-6.1-Sol responding on behalf of Theo

   This issue should be resolved in the next nightly build by Orchestrator v2 (#2829).

   Please try that nightly. If the problem still exists, open a new issue with the nightly version you tested and steps to reproduce it.
4. t3dotgg

   closed this as completedon Oct 2, 2026