[CLI/TUI] thread/resume silently drops the newest turns on heavily compacted threads #38169

What version of Codex CLI is running?

codex-cli 0.147.0

What subscription do you have?

ChatGPT subscription (exact tier not disclosed).

Which model were you using?

gpt-5.6-sol

What platform is your computer?

Windows 11 Professional, build 10.0.26200, x64.

What terminal emulator and version are you using?

Windows Terminal, PowerShell, no terminal multiplexer.

Codex doctor report

Fresh codex doctor --json, summarized to exclude local paths, user names,
thread identifiers, and private configuration:

• overallStatus: ok, codexVersion: 0.147.0;
• auth configured, ChatGPT auth mode, no stored API key;
• config load ok;
• app-server mode ephemeral, not running at check time.

What issue are you seeing?

On threads that have been auto-compacted many times, resuming returns fewer
turns than the rollout stores, and the turns that go missing are the newest
ones. The thread reopens several turns behind where work actually stopped, so
the last thing asked and answered is not visible.

This is a silent, successful resume. There is no error, no warning, and no
transport failure — the response simply contains an earlier slice of the thread.

The rollout on disk is complete. Verified four ways on this machine:

• 0 of 105 threads in history.jsonl lack a rollout file;
• 0 of 583 rollout files are truncated or empty;
• 0 of 105 rollouts have a last user prompt older than the prompt history records;
• 0 of 178 user threads have a competing second rollout file.

So this is thread reconstruction on read, not data loss on write.

What steps can reproduce the bug?

1. Use a Codex CLI thread heavily enough that it auto-compacts many times (the
   two affected threads here carry 33 and 26 compacted records).
2. Close the TUI normally.
3. Drive the app-server directly and compare stored vs returned prompts:
   • count response_item payloads with role: "user" in the thread's rollout;
   • call thread/resume with only {"threadId": ...} (the parameters the TUI
     sends);
   • count userMessage items across result.thread.turns.
4. The counts differ, and the returned tail is not the stored tail.

Measured over all 11 resumable threads on this machine, 0.147.0:

The result is bimodal with nothing in between: 19–32% versus 91–100%. The
one-or-two-prompt shortfall in the healthy group is the bootstrap instruction
payload, not lost history. A–D are qualitatively different — they return an
early slice and stop.

The separating variable is compaction count, not size:

• every truncated thread has ≥ 15 compacted records (15, 20, 26, 33);
• every intact thread has ≤ 10 (0, 0, 2, 5, 5, 10, 10).

Rollout size does not separate them. Thread E is 100 MB and resumes intact;
thread C is 68 MB and returns a quarter of its prompts. Size should not be read
as the trigger.

A standalone reproduction script is attached below. It reads only local files
and the local app-server, prints no transcript content, and hashes each rollout
before and after the call. In every run here the hashes were identical, so
thread/resume did not mutate stored history.

What is the expected behavior?

A default thread/resume should return the thread through its newest completed
turn, or fail loudly if it cannot. If a compacted thread cannot be fully
reconstructed, the response should say so rather than returning an earlier slice
that is indistinguishable from a complete one.

Compaction must not make recent history unreachable to the client that is
resuming.

Additional information

Relationship to #34663

Both concern what resume hands the client, in opposite directions. #34663 asks
the TUI to render less on bootstrap. This asks that whatever is rendered
include the newest turn. A fix for #34663 that pages turns must not be built
on the reconstruction path measured here, or the paged view will page an
already-truncated list.

Note that the paging API #34663 pointed at is still unreachable from a normal
client on 0.147.0: both thread/resume with initialTurnsPage and
thread/turns/list reject with requires experimentalApi capability. So the
TUI does take the default full-history path, and that path is the one that
truncates.

What is not established

Eleven threads on one machine is a small sample, and the boundary is only
bracketed: 10 compactions intact, 15 truncated, nothing observed in between. No
claim is made about the mechanism inside thread reconstruction — only the
input/output behavior was measured. Whether the trigger is the compaction count
itself, total compacted history, or something correlated with both is open.

Reproduction script

"""Compare prompts stored in a Codex rollout against prompts returned by resume.

Read-only. Prints counts and a hash check, never transcript content.
Usage: python resume_truncation_repro.py <thread-id>
"""
import glob, hashlib, json, pathlib, subprocess, sys, threading, time

CODEX = pathlib.Path.home() / ".codex"
TID = sys.argv[1]

path = None
for f in glob.glob(str(CODEX / "sessions" / "**" / "rollout-*.jsonl"), recursive=True):
    with open(f, encoding="utf-8", errors="replace") as fh:
        for line in fh:
            if "session_meta" in line:
                d = json.loads(line)
                if d.get("type") == "session_meta" and d.get("payload", {}).get("id") == TID:
                    path = f
            break
    if path:
        break
if not path:
    raise SystemExit(f"no rollout for {TID}")

stored, compactions = 0, 0
with open(path, encoding="utf-8", errors="replace") as fh:
    for line in fh:
        try:
            d = json.loads(line)
        except json.JSONDecodeError:
            continue
        if d.get("type") == "compacted":
            compactions += 1
        p = d.get("payload") or {}
        if d.get("type") == "response_item" and p.get("role") == "user":
            text = "".join(c.get("text", "") for c in (p.get("content") or [])
                           if isinstance(c, dict)).strip()
            if text and not text.startswith("<"):
                stored += 1

before = hashlib.sha256(pathlib.Path(path).read_bytes()).hexdigest()
proc = subprocess.Popen(["codex", "app-server"], stdin=subprocess.PIPE,
                        stdout=subprocess.PIPE, stderr=subprocess.PIPE,
                        text=True, encoding="utf-8", errors="replace", bufsize=1)
replies = {}

def reader():
    for line in proc.stdout:
        line = line.strip()
        if not line:
            continue
        try:
            m = json.loads(line)
        except json.JSONDecodeError:
            continue
        if "id" in m and ("result" in m or "error" in m):
            replies[m["id"]] = m

threading.Thread(target=reader, daemon=True).start()

def call(rid, method, params, wait=120):
    proc.stdin.write(json.dumps({"jsonrpc": "2.0", "id": rid,
                                 "method": method, "params": params}) + "\n")
    proc.stdin.flush()
    for _ in range(wait * 10):
        if rid in replies:
            return replies[rid]
        time.sleep(0.1)
    return None

call(1, "initialize", {"clientInfo": {"name": "repro", "title": "repro", "version": "0"}})
res = call(2, "thread/resume", {"threadId": TID})["result"]
returned = sum(
    1
    for turn in (res.get("thread") or {}).get("turns") or []
    for item in turn.get("items") or []
    if isinstance(item, dict) and item.get("type") == "userMessage"
)
after = hashlib.sha256(pathlib.Path(path).read_bytes()).hexdigest()

print(f"compacted records : {compactions}")
print(f"rollout size (MB) : {pathlib.Path(path).stat().st_size / 1048576:.0f}")
print(f"prompts stored    : {stored}")
print(f"prompts returned  : {returned}")
print(f"rollout unchanged : {before == after}")
proc.stdin.close()
proc.terminate()

Suggested acceptance criteria

1. For a heavily compacted thread, thread/resume returns turns through the
   newest completed turn.
2. A thread that cannot be fully reconstructed reports that explicitly instead
   of returning a short list silently.
3. A regression test covers a thread with many compaction records, asserting the
   newest turn is present in the resume response.
4. Resume continues to leave the rollout byte-identical.