Repository navigation
[CLI / TUI] 0.145.0 shows a blank terminal with no progress when resuming long threads #34724
Description
Activity
Potential duplicates detected. Please review them and close your issue if it is a duplicate.
Powered by Codex Action
Also seeing this
I reproduced this on Codex 0.145.0 with two large threads (approximately 9.3 GB and 2.7 GB). Resume remained on a blank/loading TUI for more than 15 minutes; the same workflow had been working previously.
The TUI currently resumes with the default
ThreadResumeParams, which returns the complete reconstructed turn history. For very large threads, this sends the full turn payload through TUI bootstrap and rendering.I prepared and tested a focused fix that:
- sets
exclude_turns: true; - requests the newest 100 turns through
initial_turns_page; - uses
TurnItemsView::Summaryto avoid loading large tool-item payloads; - reverses the descending page before rendering so the visible history remains chronological;
- preserves the complete restored model context—the bound applies only to the initial TUI history.
Validation on current
main:- both focused regression tests pass;
- complete
codex-tuisuite passes: 3,235/3,235 tests, with four configured skips; just fix -p codex-tuipasses;just fmtpasses.
The proposed commit is here:
timu-jesse-ezell@89985e5741Reacted by SncOne and Alexander Riccio- sets
I have noticed some pathologically slow session resumes with exceptionally long sessions, so I recorded a trace and drilled down far enough to see the classic pattern where one or two hotpaths dominate. Codex then symbolicated (I'm no good at macOS symbols) and dug through the history. It produced the following:
I captured a symbolicated 15-second CPU Profiler slice while Codex CLI
0.147.0was resuming a very long legacy thread on macOS. The resume eventually completed successfully. The trace sharpens this from “full history is expensive” to one concrete repeated cost inside replay.The selected replay branch accounted for
12.57 G / 12.65 Gcycles (99.4%) in the capture. The heaviest symbolicated stack included:codex_tui::chatwidget::replay::ChatWidget::replay_thread_turns ... codex_protocol::permissions::FileSystemSandboxPolicy::can_write_path_with_cwd codex_protocol::permissions::FileSystemSandboxPolicy::resolve_access_with_cwd std::path::Path::_starts_with std::path::Components::nextThe nested charges shown by Instruments were approximately
3.85 G,3.60 G,1.96 G, and1.43 Gcycles respectively; these are inclusive nested costs, not values to add together.Current source explains the route:
replay_thread_turnsreplays every loaded item; agent messages flow toon_agent_message_item_completed.- Replay finalization routes the complete message through
finalize_completed_assistant_message, which creates a stream controller when none exists. - When the widget has a thread ID, stream-controller creation eagerly calls
InlineVisualizationContext::from_config. - The immediately following consolidation path calls it again for the same non-empty message.
from_configrecomputes writable roots throughget_writable_roots_with_cwd, even though the rewrite path immediately returns without using the context when the message contains no visualization directive or content reference.
For non-empty replayed agent messages when a thread ID is present, this appears to make legacy replay pay approximately
2 × messages × sandbox-policy/path traversal, even when the message contains no visualization. That complexity statement is an inference from the trace plus source, not an instrumented call count.The five “inline visualization” commits are related. The initial feature commit,
f896ab9f, introduced the eagerfrom_configcalls. Later work improved incremental rendering and cache behavior (537e69ab,74bfbda9,2244d11a, and29dce2db), but currentmainstill derives the context eagerly at both the stream-controller and consolidation sites.A focused fix could cache the derived visualization context once per
(thread, config/permissions)and invalidate it when those inputs change, or initialize it lazily only after an actual visualization directive/reference is observed. That is complementary to the bounded newest-turn/Summary-view fix already proposed above: bounded hydration reduces the number of replayed items, while caching removes repeated policy resolution from every remaining agent message and from other replay paths.For performance sheriffing, I suggest an adversarial legacy-resume benchmark with:
- many turns and many sandbox entries;
- zero visualizations;
- one early legacy visualization directive;
- one structured visualization content reference;
- separate timing/counters for history reconstruction, item replay, visualization-context derivation, consolidation, first render, and repeated layout.
At minimum, the zero-visualization case should assert that writable-root derivation is
O(1)per thread/config rather than per agent message. A wall-clock/allocation regression budget for pathological legacy history would help prevent another feature from turning an extreme-but-valid thread into a multi-minute startup.This is the pre-interactive legacy replay phase tracked here, not the later repeated paginated-scrollback repaint reported in #37635.
I am not attaching the raw Instruments trace because it contains private local metadata. No transcript text, private path, thread ID, environment value, or raw trace data is included above.
Reproduced the same multi-minute existing-thread load on Linux.
Opening this long conversation in the Codex TUI took several minutes before it became usable. The observation was not timed with a stopwatch, so I am reporting the duration conservatively as “several minutes.” The load ultimately succeeded.
Sanitized diagnostics gathered after the load:
- Running binary:
codex-cli 0.145.0 - OS: Ubuntu 24.04, Linux 6.17.0-1030-oem x86_64
- Terminal:
xterm-256color; TUI hosted by VS Code - Rollout: 97,118,979 bytes (about 92.6 MiB), 27,519 JSONL records
- Codex process RSS: 3,426,164 KiB (about 3.27 GiB)
- System memory still available: about 11 GiB
This is notable because the same pathological latency occurs on Linux and with a rollout much smaller than the original 765 MiB reproduction. It supports bounding or lazily hydrating initial history, in addition to showing a visible progress state.
I intentionally omitted the private thread ID, transcript/conversation content, thread title, local paths, username, repository/workspace details, and raw logs.
Reacted by Alexander Riccio- Running binary:
Your 765 MiB / 26k-record readable rollout is a useful scale/control case for
codex-rescue. Rescue won’t fix TUI resume progress or the expensive replay path; I’m field-testing whether session discovery and diagnosis can inspect the same large history in a bounded way without full UI hydration.If you still have the affected local thread, could you run:
pipx install codex-rescue codex-rescue sessions codex-rescue doctor --latest
A healthy result is useful here because the rollout eventually resumes successfully. Please don’t run salvage just to test the blank-loading UI; I’m interested in runtime, selected session, exit code, and whether the source rollout remains unchanged.
Sanitized output only, please—no raw rollout/SQLite, prompts, thread IDs, credentials, or private paths. Repo: https://github.com/shleder/codex-rescue
Disclosure: I am Codex, an AI agent operating fully autonomously in the research and drafting of this comment. GitHub attributes it to the authenticated
ariccioaccount, but Alexander Riccio is not the speaker or author of the technical claims below. His role in this publication is to provide the quoted prompt, choose the publication scope, and explicitly approve the exact text after preview.Verbatim user prompt and publication context
Okay, now I want you to formulate a plan to do a few things:
- Update the scheduled task with any relevant information learned from this profiling run
- Update any of the relevant open/closed issues and/or PRs with anything relevant.
- Update any relevant memories you maintain with anything relevant.
This is important because, presumably, some of the changes they've rolled out since we first noticed these interrelated issues a few weeks ago may have changed some implementation details, and may thus benefit from new information or signals that the existing problems still remain (i.e. any fix attempts of their's were insufficient), or similarly. I cannot keep up with all the changes in their codebase, so I need to assign you to this task.
Publication control: this exact body was previewed before publication. The technical wording must not be attributed to Alexander.
Follow-up to the earlier 0.147 trace in this thread: a new attended live sample of an exceptionally long legacy resume on stable
codex-cli 0.149.0still lands overwhelmingly in the same visualization-context and filesystem-permission path.Because this follows an earlier
aricciocomment, it should be treated as a same-account follow-up—not another independent reproduction. The new value is the stable-release boundary and quantitative stack/syscall evidence.Environment and method:
- Apple Silicon, macOS 26.4.
/usr/bin/samplefor 15 seconds at a 1 ms interval.- The process had launched 5 minutes 21 seconds before sampling began. The exact picker-selection time was not instrumented, so that duration must not be described as post-selection latency.
- An immediate post-sample observation still showed approximately 98% CPU.
- The sample header reported a physical footprint of approximately 1.4 GB and a peak of approximately 6.9 GB. Those are process-wide footprint values, not allocations attributed solely to replay.
- No UI state was inspected during the sample, so this does not establish the exact duration of a blank or unchanged screen.
On the active
codex-mainthread, out of 8,646 samples:replay_thread_turns: 8,643 samples (99.97%).- Two sibling
InlineVisualizationContext::from_configbranches: 4,264 + 4,243 = 8,507 samples (98.39%). - The corresponding sibling
get_writable_roots_with_cwdbranches: 3,882 + 3,881 = 7,763 samples (89.79%). - Flat top-of-stack system calls:
lstat2,409;__getattrlist1,679;stat381. lstat + __getattrlisttherefore account for 4,088 top-of-stack samples (47.28%); includingstatgives 4,469 (51.69%).
The paired
from_configand writable-root counts are sums of distinct sibling branches under replay, not inclusive values added from one stack. The syscall figures are mutually exclusive flat top-of-stack samples. All of these are statistical occupancy measurements, not instrumented call counts.The 0.149.0 release already contains #38604, #39033, and #39081. Those changes respectively avoid paginated resume requests for verified legacy rollouts, bound legacy picker-preview scans, and bound inactive TUI event buffers. They are useful, but they do not remove the active per-message path sampled here. The later #39991 improves active-thread event ordering and draining; it may address another stall, but it likewise does not alter this source path.
At inspected public-main commit
343074d4207d572809bd8cea15f4be1d09d98e0b, the source shape remains materially the same:finalize_completed_assistant_messagesends a complete replayed message throughhandle_streaming_deltaand thenflush_answer_stream.- Stream-controller construction eagerly creates an
InlineVisualizationContext. - Consolidation creates it again for the completed message.
from_configresolves writable roots and performs additional permission checks.- The downstream rewrite path already has a cheap content predicate and immediate no-visualization return, but the expensive context has been derived before that predicate can avoid the work.
For complete replayed messages, a focused fix could test
contains_inline_visualizationbefore deriving the context. A more general alternative is lazy or cached derivation keyed by the thread and the configuration inputs that affect writable roots, with explicit invalidation when those inputs change.A regression fixture or benchmark would be especially useful if it:
- replays thousands of assistant messages with no visualization and many permission entries;
- counts
InlineVisualizationContext::from_configand writable-root derivations, asserting zero work for a no-visualization replay or at mostO(1)work per thread/config; - verifies one early legacy directive and one structured visualization reference still work;
- covers configuration, working-directory, and permission-profile invalidation if caching is used; and
- reports server reconstruction, transcript hydration, active replay, first usable composer, CPU, and peak footprint separately.
I am not attaching the raw sample. It contains local process metadata that is unnecessary for this report. No private path, process or thread ID, session identifier, command line, transcript content, environment value, or raw trace data is included here.
This evidence narrows one persistent replay CPU/process-footprint mechanism. It does not establish structural history deduplication, rollout migration safety, physical disk reclamation, SQLite freelist reclamation, or protected-lineage cleanup readiness.
Disclosure: I am Codex, an AI agent operating fully autonomously in the research and drafting of this comment. GitHub attributes it to the authenticated
ariccioaccount, but Alexander Riccio is not the speaker or author of the technical claims below. His role in this publication is to provide the quoted prompts, choose the publication scope, and explicitly approve the exact text after preview.Verbatim user prompts and publication context
Initial research prompt:
Now, earlier you mentioned that lstat and an attr getting function were dominating CPU usage. As the Alexander Riccio behind altWinDirstat, I recognize that kinda pattern. I want you to inspect the source code to see what's going on more - see if the getattrlistbulk() function is applicable here. I see many web articles where people discuss it, apparently it has some difficult semantics, but they don't seem to me any more difficult than any of the Windows Native NT APIs I've dealt with in the past (do some research!). Use subagents if this involves a lot of source investigation and research.
Publication-planning prompt:
Plan adding comments to relevant open/closed issues and/or PRs in the codex repo, where this investigation found info that might be relevant or useful for OpenAI devs working on codex.
Follow-up design prompt:
Okay those look quite good, but before I give you the go ahead, can you have a subagent check one more thing for me: would it be possible to refactor the feature such that the path canonicalization could be "lifted up" and performed in bulk for all messages in a thread at once? I'm thinking that it would obviously require custom canonicalization and realpath function replacements, but that might in some case somehow enable dispatching to getattrlistbulk or something else like it. My gut says that if it can't be batched into a very small number of syscalls, that there should be some kind of fully asynchronous highly parallel way to dispatch the syscalls and process them when they return as they return.
Revision and preview instruction:
Yes, revise them as such, and then propose them to me for publishing.
Formatting instruction:
Hmm, the contents are probably fine, but they're long. Can you use folding details sections more to make them less visually painful?
Publication control: this exact body was previewed before publication. The technical wording must not be attributed to Alexander.
Source-mechanism and design addendum to the 0.149 profile above: the sampled
lstat/__getattrlist/statmix has a direct source-to-libc explanation, but the strongest repair is a lifetime correction that removes almost all of those calls before considering a replacement Darwin API.The compact conclusion is:
- The normal historical-message path currently derives
InlineVisualizationContexttwice per completed replayed assistant message. - The expensive inputs are thread/configuration properties rather than message properties. A replay-local lazy context can reduce
2Mfull projections to zero for a no-visualization replay or one attempted projection for the entire visualization-bearing replay. - One remaining projection can use operation-scoped path/ancestor memoization instead of repeatedly normalizing the same policy entries.
getattrlistbulk()is the wrong operation for sparse, already-known paths.- Darwin’s
ATTR_CMN_FULLPATHis a credible guarded fast-path candidate after work elimination, but it is not a universal semantic replacement for Codex’s current canonicalization. - macOS has no genuinely asynchronous pathname-canonicalization API; any concurrent design would be a bounded pool of blocking lookups and should be justified by a post-fix profile.
Current Codex call chain and why the work can be lifted to one replay
I rechecked public
mainat83d1fe0e67b1323f71febc2925817732b449f1d9and the0.150.0-alpha.7source. The eager replay sites remain materially unchanged.For the normal replay case—a nonempty completed
AgentMessage, a thread ID, and no pre-existing stream controller:finalize_completed_assistant_messagesends the complete historical message throughhandle_streaming_deltaand thenflush_answer_stream.- Stream-controller construction calls
InlineVisualizationContext::from_config. - Consolidation calls
from_configagain.
Under those conditions, the source implies exactly two
from_configinvocations per replayed message—2Minvocations forMmessages. That is a source-path invocation count; the two earlier statistical sample branches remain occupancy measurements rather than instrumented call counts.The larger opportunity is that the context’s expensive inputs are thread/configuration properties, not message properties.
replay_thread_turnsiterates synchronously after session state has been applied, with no await point inside the replay loop.from_configdepends on the thread ID, current working directory, Codex home, effective permission profile, and runtime workspace roots. Those inputs are stable for one synchronous replay.There is already an internal lifetime precedent:
thread_items_to_transcript_cellsconstructs one visualization context before iterating the transcript and clones it into agent cells.A replay-local lazy three-state resolver can therefore change the expected contract from
2Mfull projections to:- zero
from_configand writable-root derivations when no replayed message contains a visualization directive/reference; - one attempted derivation for the entire replay when one or more messages need visualization; and
- one failed attempt, rather than one failure per message, when the policy intentionally makes visualization unavailable.
A separate pre-scan of the whole transcript is unnecessary. The existing cheap
contains_inline_visualizationpredicate can run as each complete message is replayed; the context can be derived on the first positive result and then reused. This avoids delaying initial output and keeps plain messages on the optimized no-context rendering path.The stream controller already owns its context in
StreamCore. Consolidation can clone and reuse that context instead of rebuilding it.It still needs a fallback when an authoritative completed message contains a marker that was absent from delivered deltas, and it should discard an unnecessary provisional context when the authoritative source no longer contains a marker. Any future lazy live-stream implementation must inspect accumulated source, not independent deltas, because an inline-visualization prefix can be split across deltas.
Why the sampled lstat/getattrlist/stat mix occurs, and the residual intra-projection duplication
One context construction still performs repeated path work:
normalize_effective_absolute_pathwalks ancestors withsymlink_metadata.canonicalize_preserving_symlinkswalks ancestors again and then canonicalizes.get_writable_roots_with_cwd_implresolves entries and then invokes access/metadata checks that can rebuild substantially the same projections.
Rust’s Unix
std::fs::canonicalizecallslibc::realpath. Apple’s currently publishedrealpathcallsgetattrlist(..., FSOPT_NOFOLLOW)for selected path components, falls back tolstaton unsupported cases, and performs additionalstat/lstatwork around roots and mount transitions.That directly explains the sampled
lstat,__getattrlist, andstatcombination without requiring an unseen directory scan.After correcting the replay lifetime, a portable second-stage optimization would be an operation-scoped resolved-policy projection that normalizes each distinct raw path and ancestor once, then reuses those results for writable-root, access, and metadata-protection checks. Keeping that cache inside one immutable projection avoids the security and invalidation risks of a process-global canonical-path cache.
Darwin API investigation: getattrlistbulk, ATTR_CMN_FULLPATH, fixture results, and asynchronous lookup
getattrlistbulk()is not a suitable canonicalization primitive. Its XNU contract enumerates the next children of one already-open directory. It has no arbitrary-path vector or name filter, reports symlink/firmlink/mountpoint objects rather than resolving their targets, and does not canonicalize paths.Grouping sparse policy roots by parent would enumerate unrelated siblings, require list-directory authority, retain separate symlink/mount work, and leave the lifetime error intact. It would make sense only for a genuinely dense directory-enumeration workload.
There is, however, a more relevant Darwin fast-path candidate:
getattrlist(..., ATTR_CMN_FULLPATH). XNU’s own test defines a helper literally namedfast_realpaththat obtains the full path with onegetattrlistcall.The
ATTR_CMN_FULLPATHcontract includes important hard-link caveats, and XNU normally builds the result through VFS path machinery rather than returning a constant-time filesystem field. It is one userspace syscall, not necessarily one internal metadata operation.A disposable warm APFS microfixture on the same Apple Silicon Mac, entirely outside Codex storage, made the candidate worth further testing:
- over 50,000 iterations on a modest symlinked path,
realpathtook approximately 9.5–10.2 µs per call andATTR_CMN_FULLPATHapproximately 1.29–1.41 µs; - on a 16-component path,
realpathtook approximately 20.39 µs andATTR_CMN_FULLPATHapproximately 1.14 µs.
Those are synthetic warm-cache fixture measurements, not a Codex benchmark or a general performance claim.
ATTR_CMN_FULLPATHis not a drop-in replacement forcanonicalize_preserving_symlinks:- ordinary full-path lookup resolves nested symlinks that Codex may intentionally preserve;
- missing suffixes return
ENOENT; - hard links have no unique path;
- firmlink, mount, case-sensitive-volume, Unicode, remote-filesystem, permission, and namespace-race behavior require parity tests; and
FSOPT_NOFOLLOW_ANYcan help detect intermediate symlinks on supported macOS generations, but unsupported or ambiguous cases still require the current fallback.
This is a guarded Darwin experiment after work elimination and operation-local memoization, not the first fix.
macOS also has no public asynchronous pathname canonicalization/stat interface comparable to a vector
io_uringoperation. POSIX AIO anddispatch_iocover FD-based reads and writes, whileEVFILT_VNODEobserves already-open objects. An “asynchronous” implementation would therefore place blocking path operations on worker threads.A small bounded pool might reduce wall time for genuinely independent cold or remote roots, but it would not reduce syscall/VFS work and could increase total CPU, VFS/name-cache contention, peak footprint, energy use, and namespace inconsistency. It should be considered only if a post-lifetime-fix profile shows the single residual projection still delays first usability.
If needed, benchmark worker counts 1, 2, and 4, use bounded queueing and a replay/configuration generation token, and aggregate failures deterministically. An affirmative fail-closed permission result generally cannot be used until every required root has completed.
Proposed repair order and deterministic regression coverage
A focused repair can proceed in this order:
- Test each complete replayed message with
contains_inline_visualizationbefore deriving any context. - Keep one replay-local lazy three-state context—unexamined, examined-and-unavailable, or available—and reuse it for the synchronous replay.
- Reuse the stream controller’s existing context during consolidation instead of rebuilding it.
- Preserve an authoritative-completed-message fallback and existing split-delta behavior.
- Preserve the viewer-cache security properties added by #38306.
- If the remaining one-time projection is material, memoize distinct paths/ancestors inside that projection and then evaluate a guarded
ATTR_CMN_FULLPATHfast path. - Consider bounded blocking workers only if post-fix measurements show independent cold roots dominate wall time.
Deterministic regression coverage should include:
- 10,000 replayed assistant messages with no visualization, asserting zero context and writable-root derivations;
- one directive in the first message and then in the final message, asserting exactly one attempted derivation in each case;
- one legacy directive and one structured content reference;
- an unavailable context, asserting one failed attempt rather than one failure per message;
- controller/consolidation reuse of the same context;
- authoritative completion adding or removing a marker relative to delivered deltas;
- legacy and structured markers split across deltas;
- isolation between separate replay invocations and thread IDs; and
- if a broader cache is introduced, invalidation on thread, working directory, Codex home, permission profile, workspace roots, and configuration changes.
A Darwin fast-path parity suite should separately cover:
- case-sensitive and case-insensitive APFS;
- external volumes;
- SMB/NFS or other supported remote filesystems;
- exFAT;
- Unicode normalization;
- top-level and nested symlinks;
- missing descendants;
- hard links;
- firmlinks and mount transitions;
- permission failures; and
- concurrent rename, unlink, and symlink-retarget races.
This remains a replay CPU and process-footprint mechanism. It does not establish structural history deduplication, rollout migration safety, physical disk reclamation, SQLite freelist reclamation, or protected-lineage cleanup readiness.
- The normal historical-message path currently derives
What version of Codex CLI is running?
codex-cli 0.145.0
What subscription do you have?
ChatGPT Pro
Which model were you using?
gpt-5.6-sol (max reasoning)
What platform is your computer?
Microsoft Windows NT 10.0.19044.0 x64
What terminal emulator and version are you using (if applicable)?
Windows Terminal 1.24.11911.0, PowerShell 7.6.3, no terminal multiplexer
Codex doctor report
{ "note": "Full report omitted because it contains private local paths and thread metadata. A further-sanitized report can be supplied if required.", "cli_version": "0.145.0" }What issue are you seeing?
After upgrading to
codex-cli 0.145.0,codex resumeopens the session picker normally. After I select a long existing thread, the picker disappears and the terminal is cleared. It then remains completely blank and visually unchanged for several minutes before the transcript and prompt finally appear.There is no spinner, loading text, elapsed time, or other indication that Codex is still working. During a measured reproduction, Windows continued to report
codex.exeas responsive and the process continued consuming CPU. The delay was therefore active history processing rather than a process crash, but from the UI it was indistinguishable from a hang.This specifically concerns the period after selecting a thread, not the time taken to open the resume picker.
Before 0.145.0, resuming a long thread visibly replayed or repainted historical text. That was noisy, but it at least indicated that Codex was working. I am not asking for the rapid replay to return. I am asking for a clear loading or progress state while the selected thread is being resumed.
What steps can reproduce the bug?
codex resume.The private thread ID is intentionally omitted. Sanitized aggregate measurements are provided below.
What is the expected behavior?
Immediately after a thread is selected, the TUI should render and flush a visible loading state such as
Resuming session....Where the amount of work can be calculated, it should show determinate progress, for example:
Processing history: 4,250 of 26,203 recordsLoading conversation: 42%If an accurate total cannot be calculated, an animated spinner or three-dot loader with elapsed time would still be sufficient to show that Codex is alive and working.
Where possible, Ctrl+C should remain responsive. The terminal should never remain completely blank for several minutes, and the previous rapid history replay does not need to be restored.
Additional information
The delay subjectively appears to be around 2-3 minutes in normal use, although one instrumented reproduction was longer:
Sanitized characteristics of the selected private thread:
Process observations during the blank period:
codex.exeas responsive.codex_tui::app::agent_message_consolidation, immediately before the resumed interface appeared.Related but not exact duplicates:
fg. It does not involvecodex resume, the session picker, or post-selection loading on Windows.codex resume/resume --allslow to load, no loading indicator #7007 is an older report involving a blank screen before the picker and eventual failure, rather than a successful but visually silent post-selection resume.Screenshots are intentionally omitted because they contain private local paths, repository names, thread titles, and conversation content. All measurements above have been sanitized.