Skip to content

[Bug]: a process without config.json silently rewrites the Codex catalog to native-only models #6529

Closed

Description

@lcxhh521

Client or integration

Codex App

Area

Catalog / models

Summary

Codex App suddenly showed only native OpenAI models; every routed model was gone. $CODEX_HOME/opencodex-catalog.json and models_cache.json had been rewritten in the same second to a native-only catalog (8 rows). config.toml was untouched. The running proxy still served all 33 routed and native models on /backend-api/codex/models. Codex re-read the shrunken file at its next app-server start and kept showing native models only.

The symptom matches the closed #1453, but the path is different: there the catalog was rewritten before a failing injection; here nothing fails.

I could not identify which process wrote the file, because no catalog writer logs anything. The code does allow this outcome, though, and it reproduces deterministically (below). Any opencodex process whose OPENCODEX_HOME has no config.json writes a native-only catalog into a CODEX_HOME that holds routed rows, and exits 0:

  1. loadConfig() silently returns getDefaultConfig() (only the openai forward provider) when config.json is missing. A read error also falls back to defaults. The returned object carries no "this is a fallback" marker that catalog code consults.
  2. syncModelsToCodex calls admitCodexWrite(), which refuses with authority: "config" when the config source is not file. But sync only honours a service-home refusal (src/codex/sync.ts:133), so it continues. Because the config refusal comes first in admitCodexWrite, the service-home ownership veto is never evaluated on this path either.
  3. The management/auto-refresh convergence funnel (fixedCommit in src/codex/convergence.ts) has no config-source check at all. The hourly default refresh from fix(catalog): discover new native models from the live Codex roster and refresh by default #6285 evaluates a fresh loadConfig() on every tick.
  4. The merge rule in src/codex/catalog/build-entries.ts drops every OpenCodex-authored routed row whose provider is not enabled in the driving config ("Deleted/disabled providers cannot retain OpenCodex-authored ghosts"). A defaults config therefore deletes every routed row.
  5. Nothing binds a CODEX_HOME to the OPENCODEX_HOME that injected it. The only binding is service install state, present only when a service is installed. No writer compares old vs new routed content, and no writer logs a catalog or cache write.

Reproduction

Fully isolated temp homes. Nothing outside them is touched.

R=$(mktemp -d); mkdir -p $R/home $R/ocx $R/codex
cat > $R/codex/opencodex-catalog.json <<JSON
{"models":[
 {"slug":"gpt-5.5","display_name":"GPT-5.5","description":"native","visibility":"list","priority":1,"supported_in_api":true,"context_window":272000},
 {"slug":"ark/glm-5.3","display_name":"ark/glm-5.3","description":"Routed via opencodex → ark (ark).","visibility":"list","priority":50,"supported_in_api":true,"context_window":272000}
]}
JSON
printf 'model_catalog_json = "%s/codex/opencodex-catalog.json"\nopenai_base_url = "http://127.0.0.1:10100/backend-api/codex"\n' "$R" > $R/codex/config.toml
env -i PATH=/usr/bin:/bin HOME=$R/home OPENCODEX_HOME=$R/ocx CODEX_HOME=$R/codex bun src/cli/index.ts sync; echo "exit=$?"

Result on dev (a99de42e7): exit 0, no warning about the config. ark/glm-5.3 is removed, and the catalog becomes the 8 native rows (gpt-5.5, gpt-5.6-sol/terra/luna, gpt-6-astra/sol/luna, gpt-6.1-sol). These are exactly the rows of the catalog found in the incident.

Version

The live proxy was 2.73.0. The reproduction is on dev a99de42e7.

Operating system

macOS 26 (arm64)

Provider and model

Three openai-chat providers with static model lists, plus the native openai pool. Model-independent.

Logs or error output

None. No catalog writer logs anything. The only forensic evidence was the mtimes of the two
rewritten files and the models_cache.json sentinels (fetched_at 2000-01-01T00:00:00Z,
client_version 0.0.0), which both the convergence and retained-sync writers emit.

Screenshots and supporting files

No response

Redacted configuration

Not relevant: the writer evidently ran with a config that did not contain the user's providers.

Proposed design

The catalog is a derived artifact published into a location another program reads at startup. Five invariants would keep it from being silently replaced by a projection from the wrong source:

  1. Config-backed removal. A write that removes OpenCodex-authored routed rows must be derived from the config currently on disk. Each namespace whose rows are dropped because the driving config does not enable it must also be absent or disabled in a readable config.json. Otherwise the write is refused and the previous catalog stays. Fresh installs are unaffected (nothing to remove), and so are discovery-driven removals for providers that remain enabled.
  2. One owner per Codex home. Record the injecting OPENCODEX_HOME in the Codex-home journal. Check it where the catalog write permit (K lock) is issued, so every writer is covered.
  3. Explicit teardown intent. Only restore/teardown paths may empty routed rows wholesale.
  4. Write audit. Every catalog/cache commit and every refusal appends a bounded record in CODEX_HOME: time, pid, entry point, config provenance, routed count before→after. Byte-identical rewrites are skipped.
  5. Owner self-heal. The owning service detects a published catalog that has lost routed rows it would publish, and republishes it.

I'd like to start with (1) as a focused PR, since it closes the reproduction above and the transient-read case of the hourly refresh. (4) would follow. (2), (3) and (5) touch the journal, the permit and ownership semantics, so I'd value maintainer direction on them first.

Checks

  • I searched existing issues and documentation.
  • I removed secrets, tokens, account details, request credentials, and personal data.

Activity

  1. added
    bugSomething isn't working
    catalogModel catalog, slugs, visibility, routed entries
    on Oct 3, 2026
  2. lidge-jun commented on Oct 4, 2026

    @lidge-jun
    Owner

    The deterministic missing/fallback-config overwrite reported here is fixed on dev. The landed changes reject destructive catalog removal without matching readable on-disk configuration, enforce Codex-home ownership and explicit teardown intent, add bounded write diagnostics, and provide owner recovery for a lost routed provider namespace.

    Landed PRs and commits:

    The original retained-sync and convergence reproductions are covered by focused regressions. The final recovery change also covers owner publications received while lifecycle write gates are closed. Current-head CI and Service lifecycle passed before integration.

    Closing the reported overwrite defect. This does not identify the process responsible for the original live incident or claim that every proposed extension is complete. Recovery detects a missing provider namespace, not missing individual rows while that namespace remains. Continuous Windows audit logging remains deferred: at most the first new-file event is recorded after successful hardening; existing files are skipped, including after restart. Native NTFS privacy/casing acceptance is not claimed. These limits remain documented in the linked PRs and lifecycle reference.

    This is a dev fix for the next release, not a claim that it shipped in 2.77.0.

  3. lcxhh521 commented on Oct 4, 2026

    @lcxhh521
    ContributorAuthor

    Follow-up on the open point in the closing note, which process wrote the native-only catalog: it was this repository's own test suite, run locally.

    With the catalog write audit (#6537, landed as #6560) running on my machine, a local bun test ./tests/codex-integration/ ./tests/cli/ ./tests/server/ ./tests/client/ run left these records in the real ~/.codex/opencodex-catalog-audit.jsonl. Paths are shortened, and each catalog line has a matching models_cache.json line:

    2026-10-03T18:45:55.319Z  writer=convergence intent=refresh target=catalog routed 0 -> 5  opencodexHome=$TMPDIR/opencodex-test-*/tmp/ocx-provider-reasoning-lists-BUc7ki
    2026-10-03T18:45:55.432Z  writer=convergence intent=refresh target=catalog routed 5 -> 0  opencodexHome=$TMPDIR/opencodex-test-*/tmp/ocx-provider-reasoning-lists-QurEDx
    

    Both lines carry the pid of the bun test process, and nothing in that run failed. The original incident (catalog and cache rewritten at 2026-10-02T15:34:43Z) falls inside a local suite run on the same machine (15:20 to 15:47Z), and the result has the same shape: native-only rows, with the convergence sentinels in the cache.

    The path: tests/server/management-provider-reasoning-lists.test.ts points only OPENCODEX_HOME at a temp home, and its management writes converge the catalog. When CODEX_HOME is unset in that Bun process (about 120 test files delete process.env.CODEX_HOME in a teardown branch), Codex writers resolve os.homedir()/.codex. On macOS that ignores the preload's sandboxed HOME. The real-home guard did not cover catalog, cache, journal or config.toml writes in the Codex home.

    #6591 closes that write path: in an armed test process, atomicWriteFile and K refuse the real Codex home before any write. Once a Codex home's journal records its owner, the ownership binding from #6558 is a second layer against the same kind of write.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingcatalogModel catalog, slugs, visibility, routed entries

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions