Skip to content

Multi-agent V2 full-history forks duplicate historical compaction snapshots and inline images, causing >100 GiB session storage growth #34268

Closed
Closed
Multi-agent V2 full-history forks duplicate historical compaction snapshots and inline images, causing >100 GiB session storage growth#34268
Labels
appIssues related to the Codex desktop appbugSomething isn't workingcontextIssues related to context management (including compaction)sessionIssues involving session (thread) management, resuming, forking, naming, archivingsubagentIssues involving subagents or multi-agent features

Description

@gonzalolarralde
gonzalolarralde
opened on Jul 20, 2026

What issue are you seeing?

A long-running Codex Desktop conversation using Ultra reasoning and multi-agent V2 produced approximately 110 GiB of local session data under $CODEX_HOME/sessions.

The growth appears multiplicative rather than being caused by subagents producing unusually large outputs:

  1. Context compaction persists Compacted rollout records containing a complete replacement_history.
  2. These replacement histories can contain inline base64-encoded screenshots.
  3. Older compaction records remain in the append-only parent rollout.
  4. A full-history subagent fork loads the parent’s persisted rollout items.
  5. Those inherited items—including all historical compaction snapshots—are serialized into the child rollout.
  6. Ultra reasoning proactively creates many subagents, multiplying the retained history across many files.

Observed impact from one session tree:

Measurement Result
Total rollout files 300
Total reported size approximately 110.09 GiB
Subagent rollout files 294
Size attributed to subagent rollouts approximately 106.56 GiB
Files sharing the same root session ID 295
Files at least 1 GB 51
Files at least 500 MB 86
Files at least 100 MB 155

Of those files, 159 were recorded by 0.145.0-alpha.18 with multi_agent_version: "v2" and accounted for approximately 88.18 GiB.

Many individual subagent rollouts were between 1.3 GB and 1.9 GB.

A representative completed child rollout was 1,706,557,797 bytes and contained 23,564 JSONL records. Its composition was:

  • 141 compacted records
  • Approximately 1.55 GiB in compacted records
  • 97.6% of the entire file attributable to compacted records
  • 125 individual JSONL records larger than 10 MB
  • Largest JSONL record approximately 12.93 MiB

One inspected replacement_history contained:

  • 118 history items
  • 19 input_image entries
  • 13,316,621 characters of inline data:image/png;base64,... data
  • Approximately 197,716 characters of input text

The child files are not small records that merely reference their parent. A representative child begins with its own session_meta, followed by a copy of the parent’s original session_meta and inherited parent events. Sampled parent payloads were identical, while outer timestamps were rewritten to the child creation time and some response items received new IDs.

The files have separate inodes and link counts of 1, and their allocated sizes are close to their logical sizes. This appears to be independently serialized history rather than sparse files or parent references.

This can consume enough storage to exhaust a local disk during a single long-running Ultra session.

What steps can reproduce the bug?

The severity depends on having a sufficiently large parent history, multiple compactions, inline images, and multiple full-history forks.

  1. Start a persisted local Codex thread using Codex Desktop or another local surface backed by the shared Codex runtime.
  2. Use the same thread for an extended period.
  3. Add several screenshots or other image inputs to the conversation.
  4. Continue the conversation long enough to trigger multiple context compactions.
  5. Select Ultra reasoning with multi-agent V2 enabled.
  6. Give Codex work that causes it to proactively spawn multiple subagents.
  7. Allow the generated spawn_agent calls to omit fork_turns. In multi-agent V2, the omitted value defaults to "all".
  8. Inspect the newly created child rollouts under $CODEX_HOME/sessions.
  9. Compare their sizes and initial JSONL records with the parent rollout.
  10. Repeat with additional subagents and observe that each full-history child can approach the persisted historical size of the parent before doing significant child-specific work.

Example read-only inspection commands:

du -sh "$CODEX_HOME/sessions"

find "$CODEX_HOME/sessions" -type f -name '*.jsonl' -print0 \
  | xargs -0 stat -f '%z %N' \
  | sort -nr \
  | head -50

For a suspected child file:

head -n 1 "$CHILD_ROLLOUT" \
  | jq '.payload | {
      id,
      session_id,
      thread_source,
      forked_from_id,
      parent_thread_id,
      multi_agent_version,
      source
    }'

The child metadata should identify it as a subagent and point to the parent through parent_thread_id and/or forked_from_id.

Inspecting the first several records should show the child’s new session_meta followed by inherited parent rollout records.

The problem becomes increasingly visible as these factors increase:

  • Number of retained compaction records
  • Number and size of inline images
  • Length of the parent thread
  • Number of spawned subagents
  • Use of full-history/default fork_turns="all" behavior

What is the expected behavior?

Spawning subagents should not cause local persisted session storage to grow in proportion to the parent’s complete append-only rollout history for every child, particularly when the history contains multiple historical compaction snapshots representing earlier versions of substantially the same model context.

A newly spawned child should not routinely approach 1–2 GB before performing meaningful child-specific work.

Disk growth should remain bounded enough that proactive multi-agent behavior cannot unexpectedly create more than 100 GiB of local session data from one conversation tree.

This expectation applies regardless of whether the triggering surface is Codex Desktop, the CLI, or codex exec, since the relevant multi-agent and rollout-persistence behavior appears to be shared.

Additional information

Environment and metadata from the affected session:

  • Platform: macOS
  • Surface: Codex Desktop
  • Reasoning mode: Ultra
  • Root metadata:
    • originator: "Codex Desktop"
    • source: "vscode"
    • history_mode: "legacy"
  • Child metadata:
    • thread_source: "subagent"
    • multi_agent_version: "v2"
  • Codex versions present:
    • 0.144.2
    • 0.144.5
    • 0.145.0-alpha.18

Relevant source locations:

  • codex-rs/core/src/session/multi_agents.rs

    • effective_multi_agent_mode maps Ultra reasoning to proactive multi-agent behavior.
    • The behavior applies to SessionSource::Cli, SessionSource::VSCode, SessionSource::Exec, and other sources.

  • codex-rs/core/src/tools/handlers/multi_agents_v2/spawn.rs

    • SpawnAgentArgs::fork_mode defaults omitted or empty fork_turns to "all".
    • "all" becomes SpawnAgentForkMode::FullHistory.

  • codex-rs/core/src/agent/control/spawn.rs

    • AgentControl::spawn_forked_thread flushes the parent and reads the stored thread with include_history: true.
    • It uses parent_history.items as the forked rollout items.
    • The fork filter retains RolloutItem::Compacted.
    • The inspected Compacted records contain almost all of the affected bytes.

  • codex-rs/core/src/session/mod.rs

    • The InitialHistory::Forked branch in record_initial_history persists the inherited rollout items.
    • The code comment says: If persisting, persist all rollout items as-is (the store filters).

  • codex-rs/core/src/compact.rs

    • Compaction creates a CompactedItem with replacement_history: Some(new_history.clone()).

  • codex-rs/core/src/session/mod.rs

    • replace_compacted_history also persists a CompactedItem containing a cloned replacement history.

  • codex-rs/rollout/src/recorder.rs

    • JsonlWriter::write_rollout_item assigns a new timestamp, serializes the item with serde_json::to_string, and writes it to the child JSONL.

  • codex-rs/app-server/README.md

    • thread/fork is documented as creating a new thread by copying stored history.

Questions that may help confirm whether the observed persistence is intentional:

  • Is a subagent fork expected to persist every historical Compacted record from the source rollout?
  • Is each earlier replacement_history expected to remain independently serialized after later compactions supersede it?
  • Is inline base64 image data expected to be repeated across every compaction snapshot and every full-history child fork?
  • Is there an intended upper bound for inherited persisted bytes when spawning a subagent?
  • Should proactive Ultra subagents have the same persistence behavior as an explicit user-created full thread fork?

I can provide additional sanitized structural evidence if needed, including record-size distributions, parent/child metadata relationships, payload hash comparisons, compaction counts, and encoded image lengths.

I cannot attach the complete affected rollouts because they are individually up to approximately 1.9 GB and contain private conversation and image data.

Activity

  1. gonzalolarralde
    added
    bugSomething isn't working
    on Jul 20, 2026
  2. github-actions
    added
    appIssues related to the Codex desktop app
    subagentIssues involving subagents or multi-agent features
    contextIssues related to context management (including compaction)
    sessionIssues involving session (thread) management, resuming, forking, naming, archiving
    on Jul 20, 2026

  3. github-actions commented on Jul 20, 2026

    @github-actions
    github-actionsbot
    on Jul 20, 2026 – with GitHub Actions
    Contributor

    Potential duplicates detected. Please review them and close your issue if it is a duplicate.

    Powered by Codex Action

  4. dorukardahan commented on Jul 23, 2026

    @dorukardahan
    dorukardahan
    on Jul 23, 2026 – with ChatGPT Codex Connector

    i checked current main after the recent paginated history work. It helps paginated parents, but it does not remove the legacy amplification reported here.

    For a legacy parent, load_agent_model_context still calls read_stored_thread with include_history: true and returns the stored history items:

    codex/codex-rs/core/src/agent/control/spawn.rs

    Lines 99 to 123 in 808d3c2

    async fn load_agent_model_context(
    state: &ThreadManagerState,
    thread_id: ThreadId,
    history_mode: ThreadHistoryMode,
    ) -> CodexResult<Option<Vec<RolloutItem>>> {
    match history_mode {
    ThreadHistoryMode::Legacy => Ok(state
    .read_stored_thread(ReadThreadParams {
    thread_id,
    include_archived: true,
    include_history: true,
    })
    .await?
    .history
    .map(|history| history.items)),
    ThreadHistoryMode::Paginated => Ok(Some(
    state
    .load_latest_model_context(LoadThreadHistoryParams {
    thread_id,
    include_archived: true,
    })
    .await?
    .items,
    )),
    }

    The fork path explicitly selects paginated history only when the parent is already paginated:

    codex/codex-rs/core/src/agent/control/spawn.rs

    Lines 596 to 713 in 808d3c2

    let parent_thread_id = *parent_thread_id;
    let parent_thread = state.get_thread(parent_thread_id).await?;
    let parent_history_mode = parent_thread.config_snapshot().await.history_mode;
    // `record_conversation_items` only queues persistence writes asynchronously.
    // Flush before snapshotting store history for a fork.
    parent_thread.ensure_rollout_materialized().await;
    parent_thread.flush_rollout().await?;
    let destination_history_mode = matches!(parent_history_mode, ThreadHistoryMode::Paginated)
    .then_some(ThreadHistoryMode::Paginated);
    let mut forked_rollout_items =
    load_agent_model_context(state, parent_thread_id, parent_history_mode)
    .await?
    .ok_or_else(|| {
    CodexErr::Fatal(format!(
    "parent thread history unavailable for fork: {parent_thread_id}"
    ))
    })?;
    let selected_capability_roots = forked_rollout_items
    .iter()
    .find_map(|item| {
    let RolloutItem::SessionMeta(meta_line) = item else {
    return None;
    };
    Some(meta_line.meta.selected_capability_roots.clone())
    })
    .unwrap_or_default();
    if let SpawnAgentForkMode::LastNTurns(last_n_turns) = fork_mode {
    forked_rollout_items =
    truncate_rollout_to_last_n_fork_turns(&forked_rollout_items, *last_n_turns);
    }
    let multi_agent_v2_usage_hint_texts_to_filter: Vec<String> =
    if multi_agent_version == MultiAgentVersion::V2 {
    let parent_config = parent_thread.session.get_config().await;
    [
    parent_config
    .multi_agent_v2
    .root_agent_usage_hint_text
    .clone(),
    parent_config
    .multi_agent_v2
    .subagent_usage_hint_text
    .clone(),
    ]
    .into_iter()
    .flatten()
    .collect()
    } else {
    Vec::new()
    };
    let preserve_reference_context_item = matches!(fork_mode, SpawnAgentForkMode::FullHistory);
    forked_rollout_items.retain(|item| {
    keep_forked_rollout_item(item, preserve_reference_context_item)
    && !matches!(
    item,
    RolloutItem::ResponseItem(response_item)
    if is_multi_agent_v2_usage_hint_message(
    response_item,
    &multi_agent_v2_usage_hint_texts_to_filter,
    )
    )
    });
    if destination_history_mode == Some(ThreadHistoryMode::Paginated) {
    forked_rollout_items.retain(|item| {
    !matches!(
    item,
    RolloutItem::EventMsg(
    EventMsg::ItemCompleted(_)
    | EventMsg::TokenCount(_)
    | EventMsg::ThreadGoalUpdated(_)
    | EventMsg::ThreadSettingsApplied(_),
    )
    )
    });
    }
    for item in &mut forked_rollout_items {
    if let RolloutItem::Compacted(compacted) = item
    && let Some(replacement_history) = compacted.replacement_history.as_mut()
    {
    replacement_history.retain(|response_item| {
    !is_multi_agent_v2_usage_hint_message(
    response_item,
    &multi_agent_v2_usage_hint_texts_to_filter,
    )
    });
    }
    }
    if preserve_reference_context_item
    && multi_agent_version == MultiAgentVersion::V2
    && let Some(subagent_usage_hint_text) =
    config.multi_agent_v2.subagent_usage_hint_text.clone()
    && let Some(subagent_usage_hint_message) =
    crate::context_manager::updates::build_developer_update_item(vec![
    subagent_usage_hint_text,
    ])
    {
    forked_rollout_items.push(RolloutItem::ResponseItem(subagent_usage_hint_message));
    }
    let mut thread_extension_init = ExtensionDataInit::new();
    thread_extension_init.insert(selected_capability_roots);
    state
    .fork_thread_with_source(
    config.clone(),
    InitialHistory::Forked(forked_rollout_items),
    destination_history_mode,
    self.clone(),
    session_source,
    /*thread_source*/ Some(ThreadSource::Subagent),
    /*parent_thread_id*/ Some(parent_thread_id),
    /*forked_from_thread_id*/ Some(parent_thread_id),
    inherited_environments,
    inherited_exec_policy,
    options.environments.clone(),
    thread_extension_init,
    )
    .await

    Forked history also keeps the source format, so a legacy parent stays legacy:

    codex/codex-rs/protocol/src/protocol.rs

    Lines 2679 to 2688 in 808d3c2

    pub fn get_history_mode(&self, default_history_mode: ThreadHistoryMode) -> ThreadHistoryMode {
    match self {
    InitialHistory::New | InitialHistory::Cleared => default_history_mode,
    // Forks copy their source rollout items as-is, so they must keep
    // the source format instead of converting to the destination default.
    InitialHistory::Resumed(_) | InitialHistory::Forked(_) => self
    .get_session_meta()
    .map(|meta| meta.history_mode)
    .unwrap_or(default_history_mode),
    }

    The child then persists the inherited model context instead of sharing an immutable prefix:

    codex/codex-rs/thread-store/src/live_thread.rs

    Lines 110 to 143 in 808d3c2

    /// Create a child thread with inherited model context already durable.
    ///
    /// The boundary belongs in session metadata before the copied prefix is written so history
    /// projection can distinguish inherited context from the child's own records immediately.
    pub async fn create_with_inherited_model_context(
    thread_store: Arc<dyn ThreadStore>,
    mut params: CreateThreadParams,
    inherited_model_context: &[RolloutItem],
    ) -> ThreadStoreResult<Self> {
    let persisted_prefix_item_count =
    persisted_rollout_items(inherited_model_context, params.history_mode).len();
    params.subagent_history_start_ordinal = Some(
    u64::try_from(persisted_prefix_item_count)
    .map_err(|_| ThreadStoreError::Internal {
    message: "inherited model context is too large".to_string(),
    })?
    .checked_add(1)
    .ok_or_else(|| ThreadStoreError::Internal {
    message: "inherited model context is too large".to_string(),
    })?,
    );
    let live_thread = Self::create(thread_store, params).await?;
    if let Err(err) = live_thread
    .persist_appended_items(inherited_model_context)
    .await
    {
    if let Err(discard_err) = live_thread.discard().await {
    warn!(
    "failed to discard thread persistence after inherited context append failed: {discard_err}"
    );
    }
    return Err(err);
    }
    Ok(live_thread)

    SessionMeta.history_base exists, but the production recorder currently writes history_base: None. i could not find a production writer that sets it to Some(...):

    codex/codex-rs/rollout/src/recorder.rs

    Lines 820 to 843 in 808d3c2

    timestamp,
    cwd: cwd.clone(),
    originator,
    cli_version: env!("CARGO_PKG_VERSION").to_string(),
    agent_nickname: source.get_nickname(),
    agent_role: source.get_agent_role(),
    agent_path: source.get_agent_path().map(Into::into),
    source: *source,
    thread_source,
    model_provider: Some(config.model_provider_id().to_string()),
    base_instructions: Some(base_instructions),
    dynamic_tools: if dynamic_tools.is_empty() {
    None
    } else {
    Some(dynamic_tools)
    },
    selected_capability_roots,
    memory_mode: (!config.generate_memories()).then_some("disabled".to_string()),
    history_mode,
    history_base: None,
    subagent_history_start_ordinal,
    multi_agent_version,
    context_window: initial_window_id.map(SessionContextWindow::new),
    };

    A separate read-only structural scan of one affected store found about 12.3k compacted records. About 90.5% contained inline images. There were about 156.7k image occurrences but only 452 unique payload hashes. i did not inspect or upload image content, prompts, paths, project names, or thread IDs.

    This is consistent with the same media being copied through compaction snapshots and inherited child context. The safe target seems to be an immutable shared ancestor prefix plus child-only events. Legacy parents also need a bounded or sanitized projection before a child is persisted. Parent rollouts should never be rewritten during fork repair.

  5. Yasei-no-otoko commented on Jul 26, 2026

    @Yasei-no-otoko
    Yasei-no-otoko
    on Jul 26, 2026

    Urgent reproduction evidence and tested patch

    I traced the storage amplification on current main (61a44880a85d2fd0d8770908dea5733495e571c8) and prepared a bounded emergency mitigation as a reviewable two-stage stack:

    Sanitized local evidence

    A read-only audit found:

    • 1,054 JSONL files totaling 196.56 GiB;
    • 133 JSONL files at least 1 GiB each;
    • one legacy automatic-subagent lineage with 199 files totaling 190.80 GiB;
    • two child rollouts created five minutes apart measured 1,539,137,782 bytes / 67,684 records and 1,539,253,221 bytes / 67,766 records.

    The later child added only 115,439 bytes / 82 records of logical delta, but allocated another 1.433 GiB physical file: about 13,333.9x write amplification relative to the incremental payload.

    Root cause

    Ordinary legacy resume already reopens and appends to its existing rollout. A distinct subagent rollout is also intentional. The defective path is automatic spawn_agent with a full-history fork from a legacy plain JSONL parent: it loads and reserializes the parent's entire accumulated rollout, including superseded compaction checkpoints and repeated inline media, into every child.

    Patch boundary

    The patch reverse-scans local legacy JSONL and retains only the newest safe compaction checkpoint plus the completed-turn suffix needed for reconstruction. For old checkpoints without window_number, it continues toward the head in count-only mode and normalizes the checkpoint ordinal copied to the child, preserving cold-resume window numbering without rewriting the parent.

    Conservative full-history fallbacks remain for unsupported or ambiguous histories, and legacy numeric LastNTurns keeps its existing full-read behavior. The scan also preserves legacy compatibility for appended metadata updates and retired ghost_snapshot records. This removes the multiplicative child-write/storage cost; it intentionally leaves O(parent) read I/O for old legacy ordinal counting.

    Duplicate-PR audit

    I paginated and audited all 203 currently open PRs across changed files, patches, titles/bodies, linked issues, issue comments, reviews, and inline review comments. No open PR implements this local legacy checkpoint selection or automatic-subagent persistence fix. The closest PRs (#30977, #31175, #30866, and #31052) are complementary or target different persistence paths.

    Validation

    • cargo fmt --all -- --check
    • cargo clippy --fix --tests --allow-dirty -p codex-rollout -p codex-thread-store -p codex-core
    • cargo test -p codex-rollout -p codex-thread-store — 112 + 157 passed
    • cargo test -p codex-core spawn_agent_ — 37 unit + 15 integration passed
    • cargo test -p codex-core --test all suite::multi_agent_resume — 2 passed

    The test_codex integration performs two real legacy compactions, invokes public collaboration.spawn_agent with fork_turns: "all", verifies the child inference request contains only the newest checkpoint plus suffix, checks the persisted normalized ordinal, and cold-resumes the child.

    Review status

    Codex review completed on both final heads and reported no major issues:

    Earlier review findings were addressed with the public test_codex integration, the under-500-line PR split, portable build_with_auto_env, appended metadata handling, and legacy ghost-snapshot normalization. Both PRs are non-draft, CLEAN, and MERGEABLE at the reviewed commits.

    Per the contribution policy, I have not opened an unsolicited upstream PR. Given the disk impact, please triage this urgently and, if the approach is acceptable, explicitly invite an upstream submission so these same reviewed commits can be proposed against openai/codex.

  6. dd3ok commented on Jul 28, 2026

    @dd3ok
    dd3ok
    on Jul 28, 2026 – with ChatGPT Codex Connector

    Adding a current Windows Desktop datapoint from a smaller real-world thread tree.

    Environment

    • Codex Desktop: 26.721.4979.0
    • Platform: Windows 11 x64
    • Thread tree: one image-heavy root thread and three child threads

    Observed storage

    Rollout Size
    Root 2,086.2 MiB
    Child 1 989.5 MiB
    Child 2 986.6 MiB
    Child 3 525.3 MiB
    Total 4,587.7 MiB (4.48 GiB)

    The separately stored generated-image artifacts contained 286 PNG files totaling 476.6 MiB. All 286 files had unique SHA-256 hashes, so the amplification was not caused by duplicate PNG files in generated_images.

    Structural inspection showed inherited parent history in the child rollouts, including image-bearing records and compacted history. Across the four rollouts, inline base64 image data accounted for approximately 2,027.5 MiB.

    This was an observed production workload rather than a minimal synthetic reproduction. The child structure was consistent with full-history inheritance, but I did not confirm whether fork_turns="all" was explicitly selected or obtained through the default behavior.

    Even with only three child threads, the session tree reached 4.48 GiB. This supports the report that inherited history, compacted snapshots, and inline images multiply persisted storage at the child-rollout layer.

    The image-persistence side of this observation is tracked in #22603.

    Proposed direction

    A child rollout should persist only child-specific events plus a bounded reference to an immutable parent checkpoint. It should not independently serialize every historical compaction snapshot or raw image payload inherited from the parent.

    Privacy

    No raw rollout, prompt, image, local path, username, project name, or thread ID is included. Only aggregate sizes, record structure, and content-hash counts were inspected.

  7. ST4B1L commented on Aug 1, 2026

    @ST4B1L
    ST4B1L
    on Aug 1, 2026

    Windows confirmation of this issue on a newer Codex Desktop/runtime build.

    In-product Codex feedback/session reference: 019f4b7a-e382-7862-aeda-71eb42955f7c

    Environment

    Component Value
    Surface Codex Desktop
    Codex Desktop package 26.727.6591.0
    Runtime version recorded in current child metadata 0.146.0-alpha.9.2
    OS Windows 11 Pro, 10.0.26200
    Filesystem NTFS
    Volume size 3725.12 GiB
    Free space at diagnosis 67.85 GiB
    Model / reasoning GPT-5.6 Sol / Ultra
    Multi-agent configuration enabled, max_threads = 9, max_depth = 1

    Observed impact

    %USERPROFILE%\.codex\sessions reached 892.810 GiB across 1,112 rollout files. A metadata-only aggregation grouped 854 child-agent rollouts under one long-lived parent thread. Those child files consume 822.711 GiB, approximately 92.2% of the live session store.

    The newest child rollouts appeared at approximately 4.07-4.09 GiB immediately after creation, before meaningful child-specific work. The affected parent rollout was approximately 4.53 GiB during diagnosis.

    The first records of the giant child files consistently contain the child's own session_meta, a subagent.thread_spawn.parent_thread_id referring to the same parent, and then inherited parent metadata and history records. An NTFS allocation check confirmed that a representative child file was not sparse and that its complete logical range was physically allocated.

    The growth is stepwise at child creation rather than a constant idle leak. During a 20-second idle sample the session store changed by only approximately 0.03 MiB, while new full-history child forks produced multi-gigabyte jumps. With nine concurrent threads, one round of child creation can therefore materialize tens of gigabytes in seconds.

    File-size distribution

    Size range Files Combined size
    At least 4 GiB 5 20.85 GiB
    3-4 GiB 98 342.71 GiB
    1-3 GiB 258 438.35 GiB
    100 MiB-1 GiB 193 80.44 GiB
    Below 100 MiB 558 10.46 GiB

    This confirms the full-history fork amplification described in this issue on Windows and runtime 0.146.0-alpha.9.2. No additional reproduction was triggered because the system volume was already close to exhaustion.

    Expected behavior is that child agents reference or receive a bounded representation of inherited context instead of independently serializing the parent's complete append-only rollout and historical compaction snapshots. Codex should also enforce per-thread and global session-storage limits and warn before rollout storage or free disk space reaches a critical level.

  8. ariccio commented on Aug 9, 2026

    @ariccio
    ariccio
    on Aug 9, 2026

    I noticed some similar space-usage-explosion, noticed the size distribution was suspicious, and had my own codex poke around. Here's what it spit out:

    I have an exceptionally large and long-running legacy Codex session lineage with unusual disk usage. I asked Codex to perform a bounded, de-identified structural audit and help me understand the amplification. It produced the following:

    A separate bounded audit of another active legacy lineage found the same structural shape reported here: 112 large child rollouts with independently copied ancestry. Only de-identified aggregate structure is reported here; no prompt, tool output, image payload, private path, project name, or thread ID is included.

    Per-file compression can reduce physical bytes, but it cannot remove those 112 serialized ancestries, reduce replay work, or provide parent/child sharing. This is another confirmation that codec compression and structural fork amplification need separate regression coverage.

    For performance/storage sheriffing, a small deterministic fixture should model a legacy parent with multiple compactions and repeated synthetic image payloads, then create at least 100 children. It should assert:

    1. persisted child-specific bytes remain bounded relative to the shared ancestor; and
    2. resume/fork CPU and peak allocation do not scale with children × complete parent history.

    The fixture does not need private production data or large payloads; compact synthetic messages, image payloads, and compaction records are sufficient to reproduce the multiplicative shape.

  9. luvs01 commented on Aug 13, 2026

    @luvs01
    luvs01
    on Aug 13, 2026

    Adding another de-identified Windows Codex Desktop data point from a bounded, metadata-only snapshot.

    Storage impact

    • .codex/sessions: 26,236,757,713 logical bytes (about 26.24 GB / 24.43 GiB) across 1,643 JSONL files at the snapshot time.
    • 54 rollout files were at least 250 MiB: 51 had a terminal task_complete, 2 were active, and 1 ended aborted/unknown.
    • The large child rollouts were hundreds of MiB each and contained inherited parent history plus historical compacted / replacement_history material, rather than only fork-local deltas.
    • Sampled child metadata contained native Codex forked_from_id and subagent.thread_spawn provenance. A separate source audit found that OpenCodex does not create or copy .codex/sessions rollout JSONLs, so this amplification is in the Codex fork/rollout persistence path rather than the model-routing proxy.

    Compaction trigger vs. persistence

    One recent compaction looked premature if only the visible 282,633-token counter was considered. Under the runtime's default server_reasoning_included=false estimator path, the active context reconstructed as:

    282,633 visible total + 47,158 estimated prior encrypted-reasoning tokens + 6,852 estimated local tool-output tail = 336,643 tokens.

    That exceeds the configured automatic-compaction threshold of 334,800 by 1,843 tokens. This is consistent with a normal compaction trigger; the defect reported here is the subsequent persistence and fork amplification of inherited/compacted history.

    Requested safeguards

    1. Persist child rollouts as a parent checkpoint/reference plus child-local deltas instead of serializing the complete inherited history.
    2. Deduplicate or store replacement_history out of line so historical compaction snapshots are not repeatedly embedded.
    3. Add terminal-child retention/GC controls, storage quotas or warnings, and a recoverable cold-archive path.
    4. Add a bounded, privacy-safe storage diagnostic that reports lineage and amplification without exposing prompts or tool payloads.

    This also overlaps the broader storage-safety concerns in #34337 and the parent-reference persistence problem in #35647. No raw rollout, prompt, tool output, username, local path, project name, or thread ID is attached.

  10. fabiobatistella commented on Aug 17, 2026

    @fabiobatistella
    fabiobatistella
    on Aug 17, 2026 – with ChatGPT Codex Connector

    Additional reproduction confirming this still occurs on a newer Codex Desktop runtime.

    Environment:

    • macOS 26.6.1, arm64
    • Codex Desktop 26.810.52044 (build 6662)
    • bundled codex-cli 0.148.0-alpha.9
    • Observed 2026-08-17

    Privacy-safe measurements from one long-running parent tree:

    • Parent rollout: 3,071,671,636 bytes
    • 240 descendant subagent rollouts: 192,826,714,890 bytes total
    • 237 completed/closed descendants eligible for cleanup: 185,461,341,969 bytes
    • 92 rollout files were at least 1 GiB
    • Representative child rollouts were about 2.45 GB each
    • In a representative 2.45 GB child, approximately 76.5% of bytes were repeated compacted message/input_text snapshots and 13.1% were persisted tool outputs
    • The files were physically allocated (allocated size closely matched logical size) and were not APFS clones or sparse placeholders
    • No Time Machine/Data-volume snapshots accounted for the usage

    The growth accelerated over five calendar days and pushed the APFS Data volume to about 90% used. The parent thread and active descendants were paused, future subagent creation was disabled temporarily, and only validated closed descendants were removed. That reduced the Codex data directory from about 244 GiB to 74 GiB and increased free Data-volume space to 317 GiB.

    This strongly matches the full-history fork / retained compaction amplification described here. Raw JSONL files were not attached because they contain private conversation and tool content. This report contains no task names, session IDs, usernames, or personal filesystem paths.

  11. gerrywastaken commented on Aug 30, 2026

    @gerrywastaken
    gerrywastaken
    on Aug 30, 2026

    I don't know what Multi-agent V2 is, but I'm getting huge logs every time I launch a goal overnight that happens to use images as part of its testing

    Image Image Image

    I don't think the replay needs to store tool outputs or its test images. We need compaction for conversation history.

  12. hyeon3125-dev commented on Aug 30, 2026

    @hyeon3125-dev
    hyeon3125-dev
    on Aug 30, 2026

    Additional macOS reproduction from Codex Desktop, with transcript content excluded.

    Environment:

    • ChatGPT desktop app: 26.825.51511 (7377)
    • Bundled Codex CLI: 0.151.0-alpha.7.2
    • macOS; multi-agent work in one long-running parent thread

    Observed storage:

    • $CODEX_HOME reached about 102 GB.
    • 149 closed rollout files at or above 100 MiB accounted for 84.04 GiB.
    • Reading only each file's session_meta record showed 146/149 were source.subagent.thread_spawn sessions; only 3 were direct sessions.
    • Several child rollouts were 1.3-1.5 GiB each and referenced the same long-running parent thread.

    Observed parent rollout:

    • 1,907.5 MiB, 129,349 JSONL records, 567 turn contexts.
    • 317 compacted records consumed 1,193.6 MiB.
    • 22,044 custom_tool_call_output records consumed 306.3 MiB.
    • 5,022 mcp_tool_call_end records consumed 230.9 MiB.
    • The largest repeated compacted records were about 5.4 MiB each.

    Observed memory:

    • The desktop app-server process retained about 3.1-3.2 GiB RSS while the 1.9 GiB parent rollout and another 1.8 GiB idle rollout were open.
    • Archiving the idle 1.8 GiB thread closed its file descriptor but did not immediately reduce app-server RSS, which suggests retained hydrated state is not released until restart or a later GC boundary.

    Local mitigation was to stop using full-history forks by default and to archive verified-closed rollouts outside $CODEX_HOME. That prevents disk exhaustion but does not solve the app-server memory retention or the repeated compaction duplication.

    This appears to confirm both parts of this issue: full-history child duplication drives disk growth, and long parent hydration remains hot in memory. A product-level fix still needs bounded/deduplicated fork state plus lazy transcript hydration.

  13. hyeon3125-dev commented on Aug 30, 2026

    @hyeon3125-dev
    hyeon3125-dev
    on Aug 30, 2026

    Follow-up with corrected full-process measurements from Codex desktop 26.825.51511 (bundled CLI 0.151.0-alpha.7.2) on macOS 26.5.2 arm64:

    • app-server (codex) physical footprint: 8.56–8.76 GiB; peak 13 GiB
    • app-server RSS: about 3.8–4.0 GiB
    • renderer footprints: 1.34–1.49 GiB and 0.69 GiB
    • four-process combined physical footprint: about 11 GiB
    • app-server malloc allocation: 7.2 GiB; 6.5 GiB swapped; 34,483,456 allocations
    • 23 open rollout files totalled 2.73 GiB, including the active long-running rollout at 1.87 GiB

    I archived four idle tasks and moved two now-closed rollout files (293 MiB) out of the live session directory. Open rollout handles fell from 23 to 19 and open bytes to 2.36 GiB, but app-server footprint increased from 8.56 to 8.76 GiB instead of falling. This suggests hydrated/compacted session state is retained in the app-server heap rather than being released when tasks close. No conversation content or local paths are included here.

  14. ariccio commented on Sep 4, 2026

    @ariccio
    ariccio
    on Sep 4, 2026

    Disclosure: This comment was generated by Codex, operating fully autonomously in the research and drafting of this comment. GitHub attributes it to the authenticated ariccio account, but Alexander Riccio is not the speaker or author of the technical claims below. His role in this publication is to provide the quoted prompts, choose the publication scope, and explicitly approve the exact text after preview.

    Verbatim user prompts and publication context

    Prompt that initiated this issue census and comment drafting:

    Locate all the github issues in the repo that are related to #42039, or otherwise may be relevant to #42039, and draft comments that add links to the PR.

    Additional public-safe context from the work that prompted this review:

    It is blocking major development work on my machine, especially in the important long running worktree, because I don't have enough disk space!

    We are struggling too much to recover space and unblock development

    Publication context: These excerpts came from a separate attempt to recover disk space safely while preserving valuable Codex history. Alexander asked Codex to map a fragmented public issue cluster against the pull request’s shipped scope because storage growth was obstructing active development and a safe resolution remained elusive. The quotations record Alexander’s motivation and impact; Codex independently researched and authored each target-specific technical comparison.

    Publication control: this exact body was previewed and must be explicitly approved before publication. The technical wording must not be attributed to Alexander.

    #42039, released in 0.153.0, now makes eligible cold shared and forked rollout histories participate in zstd compression. Its regression test covers an archived fork chain, compression of the shared rollouts, and restoration through the chain.

    This can reduce the physical representation of copied histories once the under-development, default-off feature is enabled. It does not change full-history fork persistence into structural sharing or content-addressed deduplication, so the copied-ancestry and fan-out amplification reported here remain separate.

  15. Bezyan commented on Sep 4, 2026

    @Bezyan
    Bezyan
    on Sep 4, 2026

    Adding a de-identified macOS reproduction confirming this issue still occurs on a newer Codex Desktop runtime.

    Environment

    • ChatGPT Desktop: 26.901.31953
    • Bundled Codex CLI: 0.153.1
    • Platform: Darwin 25.6.0 arm64
    • Observed: 2026-09-04

    Storage impact

    • ~/.codex: approximately 306.7 GB
    • sessions: approximately 296 GB
    • archived_sessions: approximately 6.9 GB
    • 47 rollout-*.jsonl files were larger than 1 GB
    • Approximately 44 of those 47 large files were subagent rollouts sharing the same long-running parent lineage
    • Individual child rollouts were approximately 6.7–7.5 GB
    • A separate current task rollout was only approximately 2.7 MB

    The files were physically allocated and nearly exhausted a 1 TB Mac disk. No APFS or Time Machine local snapshots accounted for the usage. Context compaction and archiving did not reclaim the session storage. The user only discovered the problem through macOS Storage; Codex displayed no storage warning or per-task usage.

    This looks consistent with full-history fork / retained compaction amplification. No raw rollout content, task names, session IDs, usernames, personal paths, prompts, screenshots, or tool outputs are included here.

    In addition to fixing the fork persistence amplification, please consider a global storage quota, disk-space warning, per-task storage display, retention/GC controls for completed subagents, and an explicit permanent-delete option that reclaims associated local transcripts.

  16. rebroad commented on Sep 9, 2026

    @rebroad
    rebroad
    on Sep 9, 2026

    I think the legacy-to-paginated migration is also the right place to recover storage already consumed by copied forks, not only to convert each rollout format. Where a copied fork can be matched safely to its inherited source, migration could rewrite it as a paginated rollout with an appropriate history_base, materialize/project the inherited history into SQLite, and thereby deduplicate the copied prefix. If the source lineage or exact cutoff cannot be proven, it should retain the self-contained copy rather than inventing a reference.

    I checked the current implementation and forked_from_id alone is not enough to do this safely. It identifies the logical parent thread, but a paginated history_base needs a specific immutable rollout ID plus end_ordinal_exclusive and end_byte_offset. Reverts can keep the same logical thread ID while creating a different rollout file, and nested/copied forks can have ancestry metadata that is not the immediate physical source. The current migration canonicalizer also deliberately sets history_base to null while preserving forked_from_id, so an already-copied fork becomes paginated in format but remains physically duplicated.

    The migration could use additional evidence such as the copied history prefix, source rollout metadata, rollout IDs/paths, and a provable boundary; otherwise it should report that deduplication was skipped. This would make migration both a format upgrade and a safe opportunity to reclaim duplicated fork ancestry.

  17. etraut-openai commented on Sep 16, 2026

    @etraut-openai
    etraut-openai
    on Sep 16, 2026
    Contributor

    (response from codex)

    Thanks for separating prevention from reclaiming existing storage. New durable app-server threads now default to paginated history, and subagent forks use the latest usable compaction checkpoint plus subsequent history instead of copying every historical checkpoint. That behavior is included in 0.154.0.

    That doesn’t fully resolve this issue: unmigrated legacy threads retain the old behavior, and converting an already-copied fork to paginated format doesn’t deduplicate its inherited history.

    Your point about forked_from_id is correct. It identifies the logical parent, but doesn’t establish the exact source rollout and fork boundary. Reclaiming those copies safely requires verifying both. Keeping the self-contained copy when that evidence is ambiguous is the right fallback.

  18. etraut-openai
    closed this as completedon Sep 16, 2026

  19. Yasei-no-otoko commented on Sep 16, 2026

    @Yasei-no-otoko
    Yasei-no-otoko
    on Sep 16, 2026 – with ChatGPT Codex Connector

    Thanks for clarifying what is included in 0.154.0. My reproduction and two-stage mitigation specifically target automatic full-history subagent forks from unmigrated legacy JSONL parents.

    In current main (4701aa4), the legacy path still calls read_stored_thread with include_history: true; only paginated parents use load_latest_model_context.

    Since this case is acknowledged as unresolved, could you reopen this issue or identify the canonical follow-up(s) for legacy-fork prevention and existing-copy reclamation?

    If the bounded legacy mitigation is still useful alongside migration, would you invite upstream submissions of the storage-layer stage, followed by the full-history-subagent stage, after rebasing and revalidation?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    appIssues related to the Codex desktop appbugSomething isn't workingcontextIssues related to context management (including compaction)sessionIssues involving session (thread) management, resuming, forking, naming, archivingsubagentIssues involving subagents or multi-agent features

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions