Activity

1. SUaDtL

   added

   sev:highTribunal/triage: high severityTribunal/triage: high severity

   on Jul 22, 2026
2. SUaDtL commented on Jul 26, 2026

   

   SUaDtL

   on Jul 26, 2026

   CollaboratorAuthor

   More actions

   Feasibility spike, 2026-07-26 — run against real Codex 0.145.0

   The open question gating this issue was whether a real-host lane needs OPENAI_API_KEY, because a required check that needs a secret cannot run on fork PRs. Answer: it splits cleanly in two, and only one half needs a credential.

   Credential-free, and it works today

   Installing the checked-out plugin through a real Codex host needs no credential and no network. Verified end to end against a disposable CODEX_HOME:

   codex plugin marketplace add <path-to-checkout>     # -> Added marketplace `codearbiter`
   codex plugin list                                   # -> ca-codex@codearbiter   not installed
   codex plugin add ca-codex@codearbiter               # -> Added plugin `ca-codex`
   codex plugin list                                   # -> ca-codex@codearbiter   installed, enabled   0.3.0
   

   marketplace add accepts a local path (<SOURCE>: "a local path, owner/repo[@ref], HTTPS Git URL, or SSH Git URL"), so the marketplace source is the checkout itself — no publishing step, no network. The install is a faithful copy: .codex-plugin/plugin.json, hooks/ (including hooks/hooks.json), skills/, routines/, and includes/ all land under $CODEX_HOME/plugins/cache/codearbiter/ca-codex/0.3.0, and config.toml records [plugins."ca-codex@codearbiter"] enabled = true.

   That covers a real slice of AC-1 (a real runtime installs the checked-out plugin at a pinned version) and gives AC-3 its negative case for free — a fixture with a broken or unregistered hooks/hooks.json can be asserted against codex plugin list output without a model.

   Not credential-free: proving a hook actually fires

   AC-2 cannot be reached without a provider. A hook only runs inside a turn, and a turn needs a model. Measured: codex exec pointed at an unreachable provider (http://127.0.0.1:9/v1, the discard port, with a DUMMY env key) initialises a complete session — workdir, model, provider, approval: never, sandbox: read-only, session id, the user message — and then dies at the network boundary:

   ERROR: stream disconnected before completion: error sending request for url (http://127.0.0.1:9/v1/responses)
   

   With ca-codex installed and enabled, hooks/hooks.json registering SessionStart, PreToolUse (2), PostToolUse, and UserPromptSubmit, and the fixture repo carrying arbiter: enabled, that run produced:

   • no persona injection — zero matches for You are codeArbiter / orchestrator persona / /ca:commands in the transcript
   • no hook artifacts — no .codearbiter/.markers, no gate-events.log

   Reproduced with CODEX_HOME both inside and outside the system temp directory, so that is not the cause.

   Two confounders remain, and they must be eliminated before this counts as a finding rather than an observation:

   1. codex exec is non-interactive and may not emit SessionStart by design. Distinguishing "the host never called the guard" from "this entry point does not raise that event" needs a run that reaches a first assistant turn — i.e. a working provider.
   2. This was 0.145.0. The manifest advertises >= 0.143.0 and records live verification on 0.144.1, so some of the gap may be version drift rather than a standing defect.

   Either way it is exactly the boundary this issue names: static tests can prove the guard's answer yet miss the more dangerous outcome where Codex never calls it. I could not demonstrate the host calling it.

   Concrete evidence that AC-4's canary is justified

   wire_api = "chat" is removed in 0.145.0 — it is a hard config-load error, not a deprecation warning:

   Error loading config.toml: `wire_api = "chat"` is no longer supported.
   How to fix: set `wire_api = "responses"` in your provider config.
   

   That is upstream protocol drift breaking a previously valid configuration between the version this repo live-verified (0.144.1) and current. A non-blocking latest canary would have reported it; nothing in required CI would.

   Also worth pinning: the workstation is now on 0.145.0, while the manifest still claims live-verified on 0.144.1 and >= 0.143.0. The supported-version claim is drifting away from anything continuously tested, which is the other half of AC-1.

   Two fixture-design gotchas, measured

   • CODEX_HOME must not be under the system temp directory. Codex refuses to create its helper binaries there and warns on every invocation: Refusing to create helper binaries under temporary dir "C:\Users\brenn\AppData\Local\Temp\". Non-fatal, but a CI fixture should use a workspace-relative directory so the lane's output is not buried in warnings.
   • codex doctor is not a gate surface. It reports no plugin or hook information at all, and it exits 1 purely because no credentials are present — so its exit status cannot express "the plugin is correctly wired" in a credential-free lane.

   Recommendation: split this issue

   • Required lane, no secret — install the checkout through a real pinned Codex runtime, assert codex plugin list reports installed, enabled, <expected version>, and assert a deliberately broken hooks/hooks.json fails visibly rather than silently. Runs on forks. Closes the "no workflow installs or launches a real Codex runtime" half of the finding.
   • Separate lane, credentialed — the live enforcement proof (SessionStart injection, an allowed write, an apply_patch and a shell block through Codex itself). It needs a provider credential, so it cannot be a required check on fork PRs: pull_request_target, a self-hosted runner, or a scheduled run against main. That is a maintainer call about where a provider secret is allowed to exist, not something to decide inside this issue.
   • Non-blocking canary — latest Codex, reporting drift like the wire_api removal above.

   Splitting also narrows the version claim honestly: the required lane can pin what is continuously tested, which is the alternative AC-1 already permits.
3. SUaDtL

   added 2 commits that reference this issue on Jul 26, 2026

   feat(ci): gate ca-codex on a real Codex host install (#408, credentia…

   ...

   70a5ab6

   feat(ci): test the Codex support window at both ends, and report upst…

   ...

   11c28e8
4. SUaDtL commented on Jul 30, 2026

   

   SUaDtL

   on Jul 30, 2026

   CollaboratorAuthor

   More actions

   Three of four ACs shipped in #550, all credential-free

   AC-1 — done, via its first arm. The manifest advertised >= 0.143.0 while CI installed exactly one version, so the range's lower bound rested on one manual check from July. The required real-host lane now runs at both ends — 0.143.0 and 0.145.0, both verified installable from npm first. Narrowing the claim instead (AC-1's second arm) would have told working 0.143 installs they were unsupported.

   The contract test no longer asserts the literal "0.144.1". It derives the requirement from the workflow matrix, so bumping one without the other fails in either direction.

   AC-3 — done, and it exposed a check that agreed with itself. CODEX-HOST-HOOKS-MATCH compared the installed hook manifest to the source manifest — but the install is a fresh copy of that source, so for anything source declares the two agree by construction. Measured against a live host:

   mutant (source claims a hook the install lacks): SURVIVED (exit=0)
   

   Every plugin-root-relative script the installed manifest points at must now exist in the install. A hook the host cannot execute is indistinguishable from one that allowed the operation. Verified against codex-cli 0.145.0: 9/9 clean, and a declared-but-absent hook now dies with the missing path named.

   AC-4 — done. An advisory @latest lane runs the identical check. Awaited by ci-passed, never in required_results; failure absorbed at step level rather than a job-level continue-on-error, which would still render the check run as FAILURE (#381).

   It paid for itself on the first run: latest resolved to 0.146.0 while the required lane pins 0.145.0 and the manifest claimed live-verified on 0.144.1. Nothing in CI had reported that. It installed cleanly and still satisfies the host contract.

   AC-2 needs a decision, not more work

   The live fixture proves SessionStart context injection, skill discovery, an ordinary allowed write, an apply_patch H-18/H-05 block, and a shell H-20/H-03 block through Codex itself.

   Every one of those needs a turn, a turn needs a model, and a provider credential cannot be a required check on fork PRs. The credential-free lane documented that boundary when it shipped, and #550 does not move it.

   The open question is which trade to take, and it is a spend-and-security decision rather than an engineering one:

   1. A repository secret on a non-fork-triggered workflow (push to main, or workflow_dispatch). Real enforcement proof, but it is not a required check on PRs, so it catches regressions after merge rather than before — and it costs model spend per run.
   2. A scheduled live lane (like the Pi promotion cron). Cheapest recurring cost, same after-the-fact detection.
   3. Narrow the claim instead. State that hook firing is verified manually per release rather than continuously, and keep the runbook as the record. Honest, free, and closes this AC by scoping it rather than satisfying it.

   Leaving #408 open on AC-2. Happy to implement whichever, but it needs a ruling on the credential first.
5. SUaDtL

   closed this as completedin #551on Jul 30, 2026
6. SUaDtL

   added a commit that references this issue on Jul 30, 2026

   docs(codex): scope the live-enforcement claim to what is actually ver…

   ...

   67cbc7f