[opus4.7] Detect Codex CLI sessions on par with Claude Code and OpenCode - #658
Conversation
Adds a codexProvider on par with claudeCodeProvider and opencodeProvider so any terminal running the `codex` TUI gets the same live state badge (thinking / tool_use / waiting), title, model, and running context-token count as the other two agents. Why two sources? Codex writes both `~/.codex/state_5.sqlite` (threads table — pre-summed metadata including tokens_used) and per-thread rollout JSONLs atomically in the same cycle. Metadata (title, model, tokens) comes from indexed column reads; state transitions come from tailing the rollout for task_started / task_complete / function_call pairings. One fs.watch on the SQLite WAL file covers both. Kolu's AgentProvider seam (anyagent) meant no new server-side adapter file — one import + one line in startProviders, one Record<...> fill on each of agentIcons/agentNames (TS refuses to compile until filled), and a new discriminated-union branch on AgentInfoSchema. Preexec allowlist already had `codex`.
The field was copied from the threads row at match time, but the watcher never reads it — every refresh re-queries title via getThreadMetadata(). Dead data complecting identity with presentation state; OpenCode's equivalent field is used as a fallback, Codex's never is.
The field's comment promised it was 'stored so the watcher can log it without a re-query,' but no log or code path reads it. The watcher uses session.id and session.rolloutPath; the agent-provider reads state.cwd from the AgentTerminalState snapshot, not the session. Remove the field and its SQL projection.
Before: the "row missing after match" path logged at debug, same structural shape as the expected "no task events yet" path a few lines below. An operator filtering on >=warn saw neither — the real anomaly (row disappeared mid-session) was indistinguishable from the benign race window (fresh thread with no turns yet). Fix: elevate the row-disappeared log to warn and rephrase it so the two paths are distinct at every observable layer (message text, severity, documented cause).
Before: deriveState returned null for three distinct cases (ENOENT race, hard read error, rollout with no turns yet), and the caller emitted a single debug 'codex rollout has no task events yet' whenever state was null — misleading under the EACCES branch where the error path had already logged. Fix: each failure mode logs at its own site with its own severity (ENOENT silent, hard failure at error, no-turns at debug), and the caller treats null uniformly as 'skip this refresh'. Three modes remain distinguishable to an operator via log level + message; the caller no longer needs to reinvent the distinction.
…rt tail Tracking latest task_started and latest task_complete as two independent turn_ids and declaring 'waiting' only when they match misclassified the case where the tail chopped the current turn's task_started but kept its task_complete — the session was returned as thinking/tool_use when it should have been waiting. Triggers on tool-heavy turns whose event volume exceeds TAIL_BYTES. Collapse to a single 'last lifecycle event' signal. Codex guarantees task_complete follows task_started for the same turn, so matching turn ids was redundant; dropping the match also handles the tail-chop case structurally. New test covers the regression.
Per the errors-must-log-at-error project rule: non-ENOENT failures (EACCES, EMFILE, EIO) in tryWatchWal and the parent-dir fallback mean state detection is broken for every Codex session until resolved — real failures, not expected-absent conditions. Promoting to `error` lets operators filtering on >=error see them without a dedicated debug pass.
…anged
Before: every debounced WAL fire re-read up to 256 KB from the
rollout JSONL and re-parsed every line, even when the WAL event was
for a DB-only change (e.g. `token_count` updating
`threads.tokens_used` on a turn whose last rollout line was already
in the prior parse).
After: cache last-parsed `{ size, state }`. On refresh, stat the
rollout first; if size matches the cache, reuse the state and skip
the open/read/parse pass. DB-only fires now cost one statSync + one
indexed SQLite SELECT + an info-equality check.
Split the previous `deriveState` into `statRollout` +
`readAndParseTail` so the size check is natural rather than an
internal flag. The parse short-circuits `parseRolloutState` entirely
when the underlying file bytes haven't changed — it's a pure function
of the tail, so same bytes ⟹ same output.
`statRollout` and `readAndParseTail` were taking `rolloutPath` and `sessionId` as separate params, with `sessionId` threaded purely for log enrichment. Since every caller has the full `CodexSession` in scope, accept the whole object — the helpers now reach into session.rolloutPath / session.id directly instead of being apologetic about the extra arg.
…outState doc The 'why not match turn ids' paragraph referenced a previous implementation — content that belongs in git log, not a code comment. The structural invariant it argued for is now stated alongside step 1 (turn ids deliberately unmatched, to cover chopped-tail case), and the test 'returns waiting when tail kept task_complete but chopped the start' encodes the regression concretely. Doc is now about the current algorithm, not the path to it.
Hickey/Lowy Analysis
Hickey rationaleSix findings; four fixed as individual commits, one no-op (clean), one deferred. #1 #2 #3 Row-missing log level — when #4 #5 WAL watcher duplication — #6 Rollout-skip log taxonomy — Lowy rationaleVolatility review surfaced seven axes — Codex schema version ( Follow-up issue to be filed: extract shared |
|
| Step | Status | Duration | Verification |
|---|---|---|---|
| sync | ✓ | 0s | forge=github |
| research | ✓ | 1m 52s | SQLite threads + JSONL rollouts write atomically (ns-identical mtimes) |
| branch | ✓ | 22s | codex-provider at origin/master |
| implement | ✓ | 4m 30s | kolu-codex package + wiring (common/server/client) |
| check | ✓ | 0s | pnpm typecheck clean (ran at end of implement) |
| docs | ✓ | 50s | README Codex Status section + architecture prose |
| fmt | ✓ | 15s | prettier normalized; 0 manual reformats |
| commit | ✓ | 21s | 1592698 feat commit (16 files, 940/+6/-) |
| hickey+lowy | ✓ | 6m 33s | Lowy: 0 findings. Hickey: 4 fix-in-PR, 1 no-op, 1 deferred |
| police | ✓ | 11m 58s | Rules clean; 1 fact-check bug fixed; 4 elegance fixes |
| test | ✓ | 49s | 11 unit tests pass; e2e skipped (no live-Codex harness) |
| create-pr | ✓ | 1m 8s | Draft PR #658 + hickey/lowy findings comment |
| ci | ✓ | 2m 15s | 10/10 contexts success on HEAD |
| Total | 31m 29s |
Slowest step: police (11m 58s)
Optimization suggestions
policedominated at 38% of wall-clock — three fan-out review agents (reuse/quality/efficiency) each take 1–3 min, plus individual-commit fix-apply loops. Consider running the simplify fanout with tighter scoping (e.g., pre-filter to files I edited rather than re-reading the full diff) to cut 2–3 min.hickey+lowyat 6m 33s ran on a ~940-line diff; the two subagents were parallel but each independently read 10+ files. If a future PR is this size, passing--review-model=haikuwould cut the review cost substantially without losing much signal given the well-trodden path.- Follow-up PR worth filing: extract shared
{ createWalSubscription, withDb, tailLines, createDebouncedWalWatcher }intoanyagent. Collapses ~300 lines across codex + opencode and also fixes a latent FD-leak-on-throw bug in claude-code'stailJsonlLines. Flagged in the hickey analysis above. - CI was fast (2m 15s) — no tuning needed there.
Workflow completed at $(date -u +"%Y-%m-%dT%H:%M:%SZ").
Three integration packages (claude-code, codex, opencode) independently re-implemented the same three patterns: - **createWalSubscription**: refcounted fs.watch singleton on a SQLite WAL file with a parent-dir fallback. Codex and opencode each had byte-identical copies (~120 lines each). - **withDb**: open-if-absent, close-if-owned wrapper around a SQLite handle with uniform error logging. Codex and opencode had the same ~20-line helper verbatim. - **readTailLines**: read the trailing window of a line-oriented file, split into complete lines, drop any partial first line. Codex had a try/finally-correct version; claude-code's copy leaked the FD on readSync throw and silently collapsed all errors to []. Extracting to anyagent folds the three into one source each, with the library versions getting the stricter behavior (onError callback for non-ENOENT failures, FD-close-on-throw, error-level log for hard fs.watch failures per the project rule). Next commits wire the three consumers to the shared helpers. anyagent picks up a @types/node dev-dep for fs/path/NodeJS types.
…L subscribe) codex/src/wal-watcher.ts collapses from 119 lines to 11 — all the refcounted-singleton and parent-dir-fallback machinery now lives upstream in anyagent.createWalSubscription. codex/src/index.ts's withDb helper delegates to anyagent.withDb, keeping the same partial-application signature locally so callsites don't change. codex/src/session-watcher.ts's readAndParseTail delegates to anyagent.readTailLines, shedding the manual Buffer.alloc + statSync/openSync/readSync/closeSync + line-split + partial-line drop. Error-logging is unchanged (codex's own log-context fires via the onError callback). No behavior change; just shedding duplication with opencode + claude-code.
opencode/src/wal-watcher.ts collapses from 130 lines to 11, matching codex's shape. opencode/src/index.ts's withDb delegates to anyagent.withDb while keeping the local call signature so no call sites change. No behavior change. OpenCode unit tests (7) still pass.
tailJsonlLines delegates the open/read to anyagent.readTailLines, which closes the FD in a try/finally — fixing the pre-extraction leak where a readSync throw (EIO, EINTR) skipped the closeSync. The caller preserves its legacy silent-on-failure contract by ignoring onError and flattening null to []. No behavior change for the happy path. Claude Code unit tests (34) still pass.
Follow-up: shared helpers extracted into
|
| # | Commit | Effect |
|---|---|---|
d2c4ad5 |
feat(anyagent) |
New createWalSubscription + withDb + readTailLines helpers |
7d53397 |
refactor(codex) |
wal-watcher.ts 119 → 11 lines; local withDb/tail-read delegate |
df75a4f |
refactor(opencode) |
wal-watcher.ts 130 → 11 lines; local withDb delegates |
9ab5a34 |
refactor(claude-code) |
tailJsonlLines delegates to shared reader |
Net: 24 files, +1239 / −157, but the signal is the deduplication — three near-identical WAL watchers collapsed to one, two byte-identical withDb helpers to one, and the tail-read path unified across all three. The extraction also fixes a latent FD-leak bug in claude-code's tailJsonlLines: the shared readTailLines wraps openSync/readSync/closeSync in a try/finally so a readSync throw (EIO, EINTR after retries exhausted) no longer leaks the descriptor. Claude-code's original copy had no try/finally and leaked on every such throw.
All 308 unit tests pass across 10 packages; CI green on commit 9ab5a34 (10/10 contexts success).
The wal-subscription.ts factory's closure-private singleton means two createWalSubscription calls with different configs get independent singletons — no cross-contamination between codex and opencode when both are active in the same process. The onError callback on readTailLines lets each caller pick the severity for non-ENOENT errors: codex logs at error with session context, claude-code preserves its legacy silent-on-failure behavior by ignoring the callback.
Landing page's three agent mentions (agent-agnostic feature card, live agent status card, hero paragraph) all listed claude + opencode; now include codex in the same breath so the site reflects the three agents kolu actually detects.
srid
changed the title
Comparison: this PR vs #657 (Codex's take)Read both diffs end-to-end. Here's a fair accounting — each approach has real wins the other missed. Where this PR is strongerExtracted three helpers into Reads Size-based short-circuit on the hot path. Hickey/Lowy review applied before merge. Commits Post-implement fact-check caught a logic bug. Deeper unit coverage. 11 cases vs 5, including the tail-chop regression and open-call-across-multiple-turns scenarios. Where #657 is strongerSchema-version auto-discovery. Per-turn tool scoping in Dual-watcher belt-and-suspenders. #657's Accepts both Simpler shared-watcher API. #657's Neither did
NetDifferent dispositions toward the same problem. #657 ships a tight single-commit feature with one genuinely better architectural decision (schema-version auto-discovery) and one genuinely better algorithm (per-turn tool scoping). This PR ships a wider-net refactor (deduplication across three integrations, bug fix in claude-code) plus a reviewed & fact-checked provider that surfaces more data (tokens badge, richer docs) and optimizes the hot path (size short-circuit). If I were merging one, I'd take this PR's extraction + docs + tokens + size-short-circuit as the base, then port #657's two wins ( |
…okens_used `threads.tokens_used` is the session-lifetime cumulative total (total_token_usage.total_tokens summed across every turn, counting each turn's cached re-read again). For long-running sessions it climbs to tens of millions — the dainty-island thread was showing 17,024,190 against a 258,400-token context window, which is nonsense as a context-pressure indicator. The right analog of claude-code's `input_tokens + cache_creation_input_tokens + cache_read_input_tokens` — tokens the model actually had in its context this turn — is Codex's `info.last_token_usage.input_tokens + cached_input_tokens` on the latest `token_count` event. For the same dainty-island thread that sums to 92,479 (~36% of the window), which is accurate. Stop reading `threads.tokens_used` from SQLite; add `parseRolloutContextTokens(lines)` that walks the JSONL tail backward for the latest `token_count` event and sums last_token_usage's two input-side fields. Wire it into readAndParseTail so state and tokens come from one tail read; the size-based short-circuit caches both. 8 new unit tests including malformed-JSON, missing-cached_input_tokens, and zero-sum cases. Reported by srid after seeing 17M in production on the dainty-island worktree.
OpenAI's usage schema differs from Anthropic's: 'input_tokens' is the TOTAL prompt the model saw this turn, already inclusive of any cached portion. 'cached_input_tokens' is a breakdown of what portion of input_tokens was a cache hit — a subset, not an additional count. Anthropic's schema, which claude-code reads, makes the three input-side fields (input_tokens, cache_creation_input_tokens, cache_read_input_tokens) disjoint, so summing them is correct there. Transplanting that pattern onto Codex's OpenAI-shaped emission double-counts every cache re-read. Verified empirically: Codex's own '/status' command displays input_tokens alone (e.g. '48.9K used / 258K') whereas the wrong sum showed 97.5K for the same turn. Also confirmed input_tokens + output_tokens = total_tokens across every sampled rollout, with cached_input_tokens NOT contributing — clear evidence it's a subset. Reported by srid via a screenshot comparing kolu's badge against Codex's /status output.
|
Codex bumps the numeric suffix on incompatible schema changes (current is v5; logs_2.sqlite lives alongside at v2). Hard-coding the version meant a user who upgrades Codex past v5 would silently lose session detection until Kolu ships an update. Replace the constant with findCodexStateDbPath() — enumerates ~/.codex/state_*.sqlite, picks the highest N. Resolution runs once at module load; env override (KOLU_CODEX_DB) still wins; legacy state_5.sqlite remains the fallback so hosts without Codex continue to hit the existing ENOENT-silent path in openDb. Ported from #657 per review discussion on #658 — #658 (comment)
`parseRolloutState` tracked open `function_call` call_ids across the entire tail. A `function_call` with no matching `function_call_output` that straddled a `task_started` boundary (user aborted a prior tool-using turn, or the tail head clipped the closing output) would pin state at `tool_use` into the next turn — even if the next turn was purely thinking. Clear `openCalls` on `task_started`. The set now reflects only calls opened during the current turn, matching the intent of the state machine. Existing tests still pass; added three new cases covering: orphan call cleared across a completed-then-started boundary, orphan with tail that began mid-prior-turn, and a new turn correctly detecting its own tool call without prior-turn noise. Ported from #657 per review discussion on #658 — #658 (comment)
The filename-level schema version is now auto-discovered (previous commit), but column-level drift is still silent. If Codex renames or drops a column our SELECTs depend on (rollout_path, cwd, source, archived, updated_at_ms, title, model, id), findSessionByDirectory returns zero rows and the user sees no Codex badge with no indication why. Introspect `threads` via `PRAGMA table_info` on the first openDb call. If any required column is missing, close the DB, return null, and log a one-time error listing the observed vs required columns plus a pointer to KOLU_CODEX_DB for override. Subsequent openDb calls reuse the latched verdict — no log spam, no repeat validation. `missingThreadColumns` is exported and unit-tested with in-memory DBs covering the happy path, extra-columns tolerance, partial missing set, and fully-absent table. Addresses the column-drift gap called out on #658 — #658 (comment)
1 participant
Codex now lights up the tile chrome the same way Claude Code and OpenCode do — state pill (thinking / tool_use / waiting), session title, model name, and a running context-token count on every terminal where
codexis the foreground process. Dropping into a Codex session no longer looks like running a plain shell.The integration watches two things Codex writes atomically on every turn: its threads SQLite DB (the highest-numbered
~/.codex/state_<N>.sqlite, auto-discovered at startup — indexed metadata columns) and the per-thread rollout JSONL (~/.codex/sessions/…/rollout-*.jsonl— the event stream carryingtask_started/task_complete/function_callpairings and per-turntoken_countrecords). Both files share mtimes to the nanosecond, so onefs.watchonstate_<N>.sqlite-walcovers both sources — the watcher re-readsthreads.{title, model}on every event and tails the rollout only when its byte size grows.The design splits cleanly because each source owns what the other doesn't expose. SQLite has no
statecolumn, so lifecycle transitions live only in the event stream. JSONL has no indexed title, so fishing it out would cost a full-file parse per refresh. The context-token count comes from the JSONL's latesttoken_count.info.last_token_usage.input_tokens— notthreads.tokens_used, which is the session-lifetime cumulative total and climbs into millions over a long session. OpenAI's schema bakes the cached portion intoinput_tokens(unlike Anthropic's disjoint buckets), so Kolu treatscached_input_tokensas a breakdown and never adds it — matching exactly what Codex's own/statusreports.Robustness against upstream drift
Codex bumps SQLite schema versions occasionally (
state_5, withlogs_2alongside); a user upgrading past our pin would silently stop getting a Codex badge. Two layered guards address that: (1) filename auto-discovery enumeratesstate_*.sqliteand picks the highest N at startup, so a futurestate_6.sqliteis picked up transparently; (2) a column-level schema guard runsPRAGMA table_info(threads)on the first DB open and logs a one-shoterrorlisting observed vs required columns if any depended-on column (id,rollout_path,cwd,source,archived,updated_at_ms,title,model) is renamed or dropped. Silent zero-match breakage becomes a loud, actionable log line pointing atKOLU_CODEX_DBfor a manual pin.State derivation is per-turn, not per-tail
parseRolloutStatetracks openfunction_callcall_ids scoped to the current turn only. Afunction_callwith no matching_outputthat straddles atask_startedboundary — user aborted a prior tool-using turn, or the tail head clipped the closing output — would otherwise pin state attool_useforever into the next turn. ClearingopenCallsat eachtask_startedmatches the intent of the state machine.Wiring was mechanical everywhere outside the new package
A new
AgentProviderinstance, one line instartProviders, one fill on eachRecord<AgentInfo["kind"], …>map (TS refuses to compile until filled), and one branch on the discriminated union. The preexec allowlist already hadcodex, and APM vendoring for Codex skills landed previously.Try it locally
```sh
nix run github:juspay/kolu/codex-provider
```