Repository navigation
v1.12: Linux run never returns after the turn completes; job dies on timeout with the output file already written #150
Description
Summary
Since v1.12, a sufficiently heavy Run Codex review step never completes on Linux. (See the control run in the comments: light runs on v1.12 finish fine, and v1.11 handles the same heavy workload cleanly.) Codex finishes the turn, writes its output file, prints its final message and token count — and then the step sits idle until the job's timeout-minutes kills it. The run is billed and discarded even though the result was already on disk.
The same workflow on v1.11 (52fe01ec70a42f454c9d2ebd47598f9fd6893d56) completes cleanly in ~87s.
Timeline from one affected run
ubuntu-latest, GitHub-hosted, runner 2.336.0, openai/codex-action@v1 resolving to 86365089eb2b84e0a8fb0717b304f8bdcb13b20e (v1.12):
| Time (UTC) | Event |
|---|---|
| 06:40:48 | job starts |
| 06:40:56 | Run Codex review starts |
| 06:47:45 | last exec completes |
| 06:47:51 | final assistant message, then tokens used / 270,199 |
| 06:47:51 → 06:55:48 | nothing at all |
| 06:55:48 | job timeout-minutes: 15 deadline |
| 07:00:49 | force-killed, job cancelled |
The step produced its full result in 6m55s, then idled for 7m57s.
The tell is the missing end marker. On v1.11 the log reads:
<token count>
##[end-action id=__openai_codex-action.run_codex;outcome=success;conclusion=success;duration_ms=86604]
On v1.12 the token count is the last line the step ever emits. No end-action, no outcome.
Configuration
- uses: openai/codex-action@v1
with:
openai-api-key: ${{ secrets.OPENAI_API_KEY }}
prompt-file: <path>
output-file: <path>
working-directory: <path>
model: gpt-5.6-luna
effort: xhigh
sandbox: read-only
output-schema-file: <path>Raising timeout-minutes does not help — the step idles longer. Lowering effort does not help either; the turn was never the slow part.
Why this loses completed work
With an explicit output-file, runCodexExec passes --output-last-message <file> straight to codex exec, so the CLI writes the result itself as soon as the turn ends. In the run above the file existed at 06:47:51.
But because the step never returns, every later step in the job is skipped — including the artifact upload that hands the result to the next job. A finished, valid, schema-conforming result is thrown away. Consumers that treat step success as "a result exists" will silently produce nothing.
Suspected area
v1.12 is the first release to route Codex through the rewritten privilege-isolation path (src/dropSudo.ts +419/-20, src/runCodexExec.ts +285/-6, new src/linuxCredentials.ts), launching via:
/usr/bin/setpriv --reuid="$uid" --regid="$nobody_gid" --clear-groups \
--no-new-privs --bounding-set=-all --inh-caps=-all --ambient-caps=-all \
-- /usr/bin/env ... "$@"
exec "$@"A plausible mechanism is a descendant process surviving the turn and holding the inherited stdio pipe open, so the wrapper never observes stream close. Supporting detail: even on a clean v1.11 run, the runner's own cleanup reports leftovers —
Cleaning up orphan processes
Terminate orphan process: pid (2460) (bash)
Terminate orphan process: pid (2462) (MainThread)
That run was 87s and spawned few children, so it exited regardless. The affected run executed many subprocesses during the turn.
A second symptom from the same run may share a cause: subprocesses failing to create a temporary directory. Retracted — a v1.11 control run produces the same errors and finishes normally, so that is ordinary sandbox: read-only behaviour and unrelated to this issue. See the comment below.
Possibly related open issues: #103 (output-schema with runAsUser fails to create temporary file), #137 (failed runs leave temporary output directories behind), #108 (proxy hangs).
Two things that would help regardless of the root cause
- Bound the wait. If the wrapper waited on process exit with a timeout rather than on stdio close, a lingering descendant would cost seconds, not the whole job budget.
- Don't hide a written result. The output file is complete before the hang. Failing the step with a message naming the written file — instead of never returning — would let callers publish the result they already paid for.
Workaround
Pin to v1.11:
uses: openai/codex-action@52fe01ec70a42f454c9d2ebd47598f9fd6893d56 # v1.11Note that v1 currently points at v1.12, so anyone on the floating tag picked this up automatically.
Activity
wjarka commentedon Aug 21, 2026
Follow-up with a control run, which both narrows the report and retracts part of it.
The failure needs a heavy run, not just v1.12
I originally had no v1.11 run comparable to the failing one, so "v1.12 hangs" was confounded with "that particular review did a lot of work". I found the missing cell:
52fe01ec86365089The v1.11 heavy run executed
pytest,uv run pytest,python -m unittest, andpython -m compileall, spent 237,369 tokens over 48 tool calls, and exited cleanly — including its post step. The v1.12 failure spent 270,199 tokens on the same class of work and never returned.So v1.12 is not broken across the board. Light runs on v1.12 finish fine, which is presumably why this got through. Something about the rewritten launch path only fails to return once the turn has spawned enough subprocess work.
Caveat: n=1 in the failing cell. I have one hung run, not a reproduction I can re-trigger on demand.
Retracting the tempfile speculation
I suggested the new
--regid=<nobody> --clear-groupslaunch might explain subprocesses failing withThat is wrong, and worth striking so nobody chases it. The v1.11 control run produced 32 of the same errors and finished normally. It is ordinary
sandbox: read-onlybehaviour, unrelated to this issue and unchanged by v1.12.What still stands
The orphan-process theory in the original report is also weaker than I implied: all three clean runs — v1.11 light, v1.11 heavy, v1.12 light — report a leftover process at runner cleanup and exit anyway.
So a lingering child is normal here and is not sufficient on its own. What distinguishes the failing run is the volume of subprocess work, not the mere presence of a leftover process.
The rest of the original report is unaffected: the turn completes, the output file is already written via
--output-last-message, noend-actionis ever emitted, and the job dies on its timeout with every later step skipped.islamJamer commentedon Aug 21, 2026
This happened to me as well.
rockwotj commentedon Aug 21, 2026
We're hitting this too
uilton-oliveira commentedon Aug 21, 2026
We are hitting the same thing on v1.12, in the "runner lost communication" variant from the previous comment. We have a before/after that separates two variables currently confounded in this thread: the v1.12 upgrade and the model / reasoning effort.
Setup
A PR-review workflow that fans out into parallel matrix jobs — one
codex-actioncall per slice of the changed files, each slice on its own GitHub-hostedubuntu-latestrunner. A second, independentcodex-actioncall runs later in the finalize job (a coverage gap-fill). Private repo, so I can only share shape and numbers.No
timeout-minuteson these jobs, so the GitHub default of 360 applied.Three windows, same workflow, 260 pass jobs
We changed
model/effortat2026-08-21T11:02Z, which happens to be ~11h after thev1tag moved to v1.12 (2026-08-20T23:38Z). That leaves a clean middle window where v1.12 was already live and our config had not changed yet.52fe01ecgpt-5.6-terra/high86365089gpt-5.6-terra/high86365089gpt-5.6-luna/xhighTwo things fall out of that:
2026-08-21T02:52Z, ~3h after the tag moved — consistent with the 01:35Z first failure reported above.Failure rate tracks how much work the turn does
Within window C, re-sampled slightly later (66 pass jobs, hence the different total), survival correlates with how many files the slice carries:
In the original sample every pass with index ≥ 2 failed (0/21). The only survivors are the smallest slices — one had 1 changed file and 21 diff anchors. That matches "light runs on v1.12 finish fine" exactly.
Two details that may help narrow it
sandbox: workspace-writeis not required for the harsher variant. The previous comment suggested the escalation from "step never returns" to "runner killed" might come fromworkspace-writeplus very long turns. We get the runner-killed variant withsandbox: read-only, on workloads that ran at p50 ~2 min, max 4.4 min across 145 jobs under v1.11. So it is not exclusive to long or write-sandboxed turns — the same work that took 2 minutes on v1.11 now kills the runner.timeout-minutesset (default 360), GitHub declared the runner dead a very consistent ~64 min after the step started: 64m02s, 64m04s, 63m02s and ~64m on separate runs. The previous comment saw 237 and 363 min against a 120 min limit. Whatever that interval is, it does not tracktimeout-minutes.BlobNotFoundfrom the REST API,log not foundfromgh run view --log-failed), so there is never step output to attach.Both the matrix passes and the independent gap-fill call in the later job die the same way, so it reproduces at two unrelated points of the same workflow.
Why it costs more than the run
Because the runner dies rather than the step failing,
continue-on-error: trueon our gap-fill step does not help: the whole job goes tofailureand the step that publishes the review never runs. We had 4 PRs where the review was fully computed and then discarded, one of them after 2h09 of wall clock. Same "finished work thrown away" as the OP, one job further downstream.We are pinning to
52fe01ec(v1.11). Happy to run an instrumented comparison if it would help — we have a workload that fails ~73% of the time on v1.12 within a couple of minutes of a trigger, which should make a bisect or astrace/pscapture cheap.fix(ai-review): pin codex-action to v1.11 and codex CLI to 0.149.0
ci: bump ai-review workflow to 1.30.20260821 (codex-action pin) (#1477)
ci: bump ai-review workflow to 1.30.20260821
viyatb-oai commentedon Aug 24, 2026
this has been fixed in bf3eb2e - it will be in a mainline release this week!
mzdm commentedon Aug 26, 2026
please escalate the release, this is consuming lots of github actions minutes
ci(security-fix): pin codex-action and CLI to the ai-review versions (#…
ci(security-fix): pin codex-action and CLI to the ai-review versions
viyatb-oai commentedon Aug 26, 2026
The fix is now available in Codex CLI 0.150.0, released today.
No action version change is needed: openai/codex-action@v1 installs the latest CLI by default. If your workflow pins codex-version, update it to "0.150.0".
Please rerun the affected workflows and let us know if you still see the hang after the turn completes.
fix(ci): bump Codex CLI to 0.150.1 to unhang security review jobs (#6962
ci: pin openai/codex-action back to v1.11 (v1.12 hangs after turn com…
ci(repo): fix codex-action v1.12 hang (downgrade to v1.11), adjudicat…
ci: pin codex-action to v1.11 and cap review-job runtime
ci: pin codex-action to v1.11 and cap review-job runtime (#141)
fix(ci): pin codex-action to v1.11 instead of weakening the sandbox
fix(ci): codex-action 을 v1.11 로 되돌려 fact-check hang 을 없앤다
ci(repo): re-pin codex-action to v1.11, block future auto-bumps (supa…
[FIX] Pin codex-action to v1.11: v1.12 never exits on heavy reviews (#…
[CHORE] Rewrite the Codex review prompt
[CHORE] Rewrite the Codex review prompt (#7668)
ci: ISSUE-449 pin codex-action v1.11 and drop the in-job verifier retry
docs(adr): record the Codex gate's v1.11 pin and salvage rule (#530)
ci: harden the Codex review gate against the post-output hang (#530)
docs(adr): record the Codex gate's v1.11 pin and salvage rule (#530)
fix(ci): bump Codex CLI to 0.150.1 to unhang security review jobs (bl…