Performance in long AutoDrive Sessions

[ConstantinHvber]

What version of Code is running?

0.6.29

Which model were you using?

gpt-5.2

What platform is your computer?

Linux 6.8.0-90-generic x86_64 x86_64

What steps can reproduce the bug?

Basically every long running session over 1-2M Token is affected.
CPU usage on one core is maxxed out, other cores almost idling.
Previous performance update helped this to some degree but at around 1-2M Tokens+ it still appears every single time.

What is the expected behavior?

Work fluently up to infinite tokens

What do you see instead?

Scrolling getting stuck, agents timer no longer counting up, scheduled autodrive prompt stuck on pending, nothing happening anymore regardless of time waited.

Additional information

Will add debugging logs soon

[ConstantinHvber]

Changed the title as I just noticed it's not related to how long an individual session is, but rather how long the session file is, because resuming a long session (multimillion token) makes the CPU on one core (only one core) max out again. Interestingly after a stop and resume, even though that CPU load persists, the performance is smooth again until a few hours (with a few millions of tokens) pass

[ConstantinHvber]

My AI debugging session spit this out:

---

I noticed the slowdown is not primarily correlated with “how long the interactive session has been open”, but with how large the
resumed session data is (multi‑million‑token / very large rollout file). When resuming a very long session, CPU on exactly one core
maxes out again (sustained). However, after a stop+resume cycle, even if that one‑core CPU load persists, UI responsiveness becomes
smooth again for a few hours; then severe lag returns once the session grows further (millions more tokens).

Implication: there are likely at least two distinct phenomena:

1. A resume-time / large-history scaling cost that can peg one CPU core (single-threaded work).
2. A state-degeneration or work-amplification bug that appears over long runtimes and makes the UI feel laggy later, even if base CPU
   usage is already high (e.g., mismatch/index churn, increased persistence cadence, stdout backpressure, warning spam).

The “smooth after restart” strongly suggests some internal index/cache state becomes inconsistent over time and is temporarily repaired
by rebuilding state on resume.

---

Comment: Current SSH host measurements (Code CLI 0.6.29) — confirmed causes with no plausible alternative explanation

Platform:

• Linux ... 6.8.0-90-generic ... x86_64 GNU/Linux
• code 0.6.29

A) Confirmed: active session *.snapshot.json is repeatedly rewritten in full (~2–3×/sec)

Evidence (read-only measurements):

• code had open fds:

  • ~/.code/sessions/.../<session>.jsonl
  • ~/.code/sessions/.../<session>.snapshot.json
  • ~/.code/log/codex-tui.log

• The active snapshot file size was ~6MB and its mtime was updating roughly every 0.4–0.6s.

• /proc/<pid>/io write_bytes increased in fixed steps of ~6,094,848 bytes, matching the snapshot file size.
  That pattern indicates the full snapshot file is being rewritten repeatedly (not small appends), which is consistent with a
  serialize-entire-snapshot + overwrite behavior.

This is a direct “smoking gun” because the step size matches the snapshot size and happens repeatedly at the same cadence as mtime ticks.
There isn’t a plausible alternative explanation (e.g., unrelated disk writes) that would produce repeated equal-size write_bytes deltas
aligned with stat mtime updates on the specific snapshot path.

B) Confirmed: this is driven by TUI history mutations (tool cards / agent status / progress) and a hardcoded snapshot flush cadence

Relevant code path in this repo (OSS source):

TUI side:

• ChatWidget::history_replace_at always calls mark_history_dirty():
  • code-rs/tui/src/chatwidget.rs:10597
• mark_history_dirty() immediately triggers snapshot flush scheduling:
  • code-rs/tui/src/chatwidget.rs:11938
  • flush_history_snapshot_if_needed(false) is throttled at 400ms:
    code-rs/tui/src/chatwidget.rs:11951
• The TUI serializes the entire HistorySnapshot and sends it to core:
  • code-rs/tui/src/chatwidget.rs:11956 (build snapshot)
  • code-rs/tui/src/chatwidget.rs:11957 (serialize via serde_json::to_value)
  • code-rs/tui/src/chatwidget.rs:11961 (send Op::PersistHistorySnapshot)

Core side:

• Op::PersistHistorySnapshot is handled in core and routed to the rollout recorder:
  • code-rs/core/src/codex.rs:5258
• write_snapshot() uses tokio::fs::write on the full JSON blob (full rewrite):
  • code-rs/core/src/rollout/recorder.rs:555

Tool-card update path (one major producer of frequent history_replace_at calls):

• tool_cards::replace_tool_card calls chat.history_replace_at(...):
  • code-rs/tui/src/chatwidget/tool_cards.rs:150

Agent progress emission can be very frequent in core:

• AgentManager::add_progress() calls send_agent_status_update() on every progress message:
  • code-rs/core/src/agent_tool.rs:783 .. :797

Result:

• Even if only a small visible status line changes, the current design can still:
  • mark the whole history “dirty”
  • serialize the entire snapshot
  • rewrite the entire snapshot file
    at up to ~2–3 times per second.

This is independent of any “sliding window” / viewport-only renderer optimization, because persistence is full-history work.

C) Confirmed (intermittent): stdout/PTY backpressure can defer redraws (UI “feels frozen”)

Observed warnings in codex-tui.log:

• stdout not writable; deferring redraw to avoid blocking

Source:

• code-rs/tui/src/app.rs:1349 .. :1396 checks stdout_ready_for_writes() and defers redraw if poll(POLLOUT) says not writable.
• code-rs/tui/src/tui.rs:327 implements readiness via poll.

This can cause “lag” symptom even if internal state updates are occurring; redraw frames are deferred to avoid blocking on stdout.

---

Comment: Why a “sliding window renderer” upgrade may not fix this

I saw a statement like:

“MAJOR upgrade to TUI performance … sliding window for rendering … cached data outside the window”

Even if that optimization works as intended for render/layout costs, the confirmed regression above is dominated by work outside the
viewport-rendering pipeline:

• full-history snapshot serialization (TUI) and full-file snapshot rewrite (core)
• history state mutations that trigger mark_history_dirty regardless of visibility
• intermittent terminal backpressure deferring redraw

So it’s possible for render performance to be significantly improved while overall UX still lags due to persistence/churn.

---

Comment: Secondary but important suspected issue (explains “smooth after resume then degrades”)

The log contains large volumes of:

• history-state mismatch: unable to locate cell for id HistoryId(...)

This warning is emitted here:

• code-rs/tui/src/chatwidget.rs:10230 (else branch of update_cell_from_record)

The lookup path that precedes it can degrade to O(n) scans over the entire history:

• cell_index_for_history_id scans history_cell_ids (.position) and may fall back to scanning history_cells:
  • code-rs/tui/src/chatwidget.rs:10189

Hypothesis (needs maintainer confirmation): as tool-card/agent updates mutate history over long runs, some mapping between
history_state records and history_cells / history_cell_ids becomes inconsistent; then every subsequent update can become more
expensive (full scans, repeated invalidations) and/or fail and log warnings. A stop+resume rebuilds structures from persisted snapshot
and temporarily restores consistency → smooth UI for hours until the bug is re-triggered.

This hypothesis is consistent with:

• “resume huge session pegs one core” (large rebuild cost)
• “smooth after resume despite CPU” (fast paths restored)
• “later degrades again” (state becomes inconsistent again)

If true, this is likely an additional issue separate from snapshot rewrite cadence; it may warrant its own bug report if reproducible.

---

Comment: Agent-status tracker churn / placeholder trackers (previously observed; still worth checking on new versions)

In some runs (older measurements), codex-tui.log was dominated by:

• status_update received with no existing tracker; creating placeholder

Source:

• code-rs/tui/src/chatwidget/agent_runs.rs:1015

The status update grouping uses a Vec + .find() which is O(n²) in number of distinct batch IDs per update:

• code-rs/tui/src/chatwidget/agent_runs.rs:962 .. :966
  And the TUI prunes agent runs beyond MAX_AGENT_RUNS = 48, potentially causing churn:
• code-rs/tui/src/chatwidget/agent_runs.rs:38
• code-rs/tui/src/chatwidget/agent_runs.rs:1621

If batch IDs exceed this limit or mappings are dropped while updates continue, placeholders/warn spam can dominate.

---

Comment: Minimal repro + “proof pack” commands (read-only)

To make the diagnosis unambiguous, capture two 60s windows:
(A) right after resume (“smooth but 1-core hot”), and (B) later when lag returns.

Replace <PID> and <SNAPSHOT_PATH>:

1. CPU/thread snapshot:

• top -H -b -n 1 -p <PID> | head -n 80
• ps -Lp <PID> -o pid,tid,stat,psr,pcpu,etime,comm | head -n 80

2. Snapshot rewrite cadence + write amplification:

• bash -lc 'pid=<PID>; snap=<SNAPSHOT_PATH>; for i in $(seq 1 20); do t=$(date +%s.%N); m=$(stat -c %y \"$snap\"); wb=$(awk '\''$1==\"write_bytes:\"{print $2}'\'' /proc/$pid/io); echo \"$t | $m | write_bytes=$wb\"; sleep 0.5; done'
  Expected bad pattern: mtime ticks every ~0.4–0.6s and write_bytes increases in ~snapshot_size steps.

3. Log symptom rates (per-second):

• Count occurrences over the last N lines and compute time span between first/last timestamps:
  • history-state mismatch
  • stdout not writable; deferring redraw
  • status_update received with no existing tracker

---

Comment: Suggested fixes (code-level; not user workarounds)

Fix 1: decouple volatile UI updates from persisted full-history snapshot

Goal: tool-card/agent progress updates should not force full snapshot persistence.
Options:

• Persist only on semantic boundaries (user turn submit, assistant final, exec completion), not on “progress ticks”.
• Separate persisted “conversation history” from volatile in-flight UI state (spinners/progress) so frequent updates do not dirty the
  persisted snapshot.

Fix 2: coalesce snapshot persistence more aggressively and/or make it incremental

Current behavior:

• throttle 400ms in TUI (code-rs/tui/src/chatwidget.rs:11951)
• full JSON serialization in TUI and full-file overwrite in core (code-rs/core/src/rollout/recorder.rs:555)
  Options:
• “latest-wins” debounce: serialize/persist at idle or on a slower cadence (seconds), cancel/overwrite pending snapshot jobs.
• Skip persistence if snapshot content hash/fingerprint unchanged (especially for progress-only updates).
• Move snapshot serialization off the UI thread (spawn task) and keep UI thread cheap.

Fix 3: resolve history-state mismatch / mapping inconsistency (likely separate bug)

Goal: updating a cell by HistoryId must be O(1) and reliable.
Options:

• Maintain a HashMap<HistoryId, usize> index for cells; update it on insert/remove/replace.
• Add invariants/tests around tool-card insert/remove/replace paths so history_cell_ids and history_state cannot diverge.
• Reduce warning spam and ensure the warning indicates a real bug (not “expected during prune”).

Fix 4: reduce agent-status churn cost

Options:

• Replace Vec grouping + .find() with HashMap<String, Vec<AgentInfo>> in handle_status_update.
• Avoid placeholder creation + warn spam when a tracker was pruned; dedupe warning per batch ID and/or ignore updates for pruned batches.

---

[ConstantinHvber]

Hi I tried around a little bit and i found a fix that works for me:

main...ConstantinHvber:code:fix-sluggishbehavior

#486