Skip to content

fix(review): give review legs an owner instead of an observer (#590) - #595

Merged
BaseInfinity merged 3 commits into
mainfrom
fix/590-dead-leg-detection
Aug 14, 2026
Merged

fix(review): give review legs an owner instead of an observer (#590)#595
BaseInfinity merged 3 commits into
mainfrom
fix/590-dead-leg-detection

Conversation

@BaseInfinity

Copy link
Copy Markdown
Owner

Closes #590.

The failure

codex exec reads stdin to EOF and appends it to the argv prompt — argv and stdin are concatenated, not alternatives. Handed a pipe nobody closes, it blocks on that read before contacting the model, forever.

On 2026-08-13 two review legs were waited on for 51 and 28 minutes. The loop's "still working, or done?" decision was made twice against a process that had never started. #341 added < /dev/null to the documented invocations in May and the hang still happened, because the leg that hung was typed ad hoc.

Two attempts to detect it, both falsified by running code

Sol (GPT-5.6 high) did not argue with either design — it broke them.

Attempt 1, watch the output file and the process table:

Signal Why it fails
tokens used marker required -o writes only the last agent message and carries no such block → a successful review reported dead
tokens used marker present it appears in codex's echoed prompt → a crashed leg reported complete
fd 0 reported as PIPE says nothing about a pending read; a pipe at EOF is healthy → called dead
39-byte hang signature machine-specific — measured at 143 bytes elsewhere, a permissions warning ahead of the notice
enumerate processes holding the file counts readers → a tail -f flipped the verdict

Attempt 2, record the child's pid and exit status in sidecars for a waiter to read: the child exits before the status is published, so a successful leg read as dead; killing only the launcher left a status that could never arrive while the child pid still looked alive; a stale status was read as this run's; a reused pid was indistinguishable from the original.

Both are the same mistake in different clothes: a second observer reconstructing what the first process already knows.

What shipped

One script, no observer. scripts/run-review-leg.sh gives the child < /dev/null — whatever it inherited itself — and execs codex, so the caller sees codex's own exit status with nothing in between that could die separately. Run as a background task, that status is the verdict.

The sidecars went with the waiter: with no reader they were only a way to be stale.

Verification

  • Live, against real codex 0.147.0, in the shape that reproduces the hang: the launcher's own stdin on an unclosed fifo. The leg completed, exit 0.
  • This PR's own round-2, -3 and -4 review legs were launched through it.
  • 9 tests, driven by a stub codex on PATH that reads stdin to EOF exactly as the real one does — so any test that finishes at all is proof the launcher supplied that EOF. No fixture pretends to be a blocked read; the previous suite passed 5/5 against fixtures that were wrong about the very thing they existed to catch.
  • shellcheck clean; doc-consistency 137/0; cowork-drift 30/0.

Review record

Four rounds, findings tapering 5 P1s → 4 P1s → 1 P1 → 0. Final: CERTIFIED 10/10, confidence 99.

Rounds 2 and 3 each asked the reviewer, on a sub-7 score, to name the smallest change that would make it pass. Both times it did, and both times that prescription — not the defect list — is what got implemented. Round 3's fix was two sentences and no code.

Known obligation, filed not closed

Consumers do not receive the launcher: scripts/ does not ship, confirmed via npm pack --dry-run. The shipped skill therefore describes a mechanism it does not supply — the #573 class. Filed as #594 with the reviewer's ship decision quoted verbatim, to land before or with the next release. It was ruled non-blocking for this PR; it is recorded as open, not resolved.

@BaseInfinity

Copy link
Copy Markdown
Owner Author

CROSS-MODEL-CLEARANCE

Sol (GPT-5.6 Sol, high) — round 4 of 4. Findings by round: 5 P1s, 4 P1s, 1 P1, 0. Verbatim: "1. Yes. The diff exactly satisfies the named certify condition. 2. VERDICT: CERTIFIED — 10/10."

{
  "confidence": 99,
  "reviewer": "gpt-5.6-sol-high",
  "sha": "9f67c9b9d6930dc40b46a85ad5a09add2cd38260",
  "verdict": "YES"
}

@BaseInfinity

Copy link
Copy Markdown
Owner Author

CROSS-MODEL-CLEARANCE

Fable (high) — design authority across all four rounds. Ruled twice against patching: round 1 that observing an already-running leg is intrinsically unsound, round 2 that the reviewer's smallest-change prescription should be taken verbatim and the sidecars dropped with the waiter. Grounds for clearance: the final design eliminates the failure class rather than detecting it; live verification against real codex in the exact hang shape; this PR's own review legs ran through the launcher; all gate-relevant suites pass; the one open item (#594) is filed, quoted, and correctly non-blocking.

{
  "confidence": 97,
  "reviewer": "fable-high",
  "sha": "9f67c9b9d6930dc40b46a85ad5a09add2cd38260",
  "verdict": "YES"
}

test-workflow-triggers.sh requires CONTRIBUTING.md to name every script CI
validate runs. The new step was added to ci.yml without it, so CI failed on
its own consistency guard.
@BaseInfinity

Copy link
Copy Markdown
Owner Author

CROSS-MODEL-CLEARANCE

Sol (GPT-5.6 Sol, high) — round 5, re-bound to the new head after the CONTRIBUTING.md CI fix. Verbatim: "No—the delta does not affect the verdict. VERDICT: CERTIFIED". Supersedes the clearance bound to 9f67c9b. CI validate green (run 31756006876, headSha 8c8f49a).

{
  "confidence": 100,
  "reviewer": "gpt-5.6-sol-high",
  "sha": "8c8f49a337832d9ef2e0f5e47ef03e5651bc81d0",
  "verdict": "YES"
}

@BaseInfinity

Copy link
Copy Markdown
Owner Author

CROSS-MODEL-CLEARANCE

Fable (high) — re-bound to the new head. The delta since the prior clearance is one line in CONTRIBUTING.md, added because the repo's own consistency guard requires every CI-run script to be listed there. It changes no behaviour and the guard that demanded it now passes. Prior grounds stand.

{
  "confidence": 97,
  "reviewer": "fable-high",
  "sha": "8c8f49a337832d9ef2e0f5e47ef03e5651bc81d0",
  "verdict": "YES"
}

@BaseInfinity

Copy link
Copy Markdown
Owner Author

DUAL CROSS-MODEL CERTIFIED MERGE

Cleared by: gpt-5.6-sol-high (posted by @BaseInfinity), fable-high (posted by @BaseInfinity) — both bound to 8c8f49a337832d9ef2e0f5e47ef03e5651bc81d0

Merge-evidence path(s) this authorised:

  • .github/workflows/ci.yml (merge-evidence path)

Verified: CI validate green across every run of that name; no net-removed test files; no package.json version bump; clearance artifact CERTIFIED at round >= 2 bound to this SHA; the executing merge script and its redirect hook byte-match origin/main.

ATTESTED, NOT AUTHENTICATED. Both clearances were posted by the same gh token, so this records that two distinct reviewers returned YES at >=95 — not that two independent principals did. A new workflow file can still mint a green required check; the compensating layer is that both reviewers read this diff.

Posted by scripts/merge-pr.sh --dual-certified before merging.

@BaseInfinity
BaseInfinity merged commit e6d2740 into main Aug 14, 2026
4 checks passed
BaseInfinity added a commit that referenced this pull request Aug 14, 2026
Post-merge audit of PR #595 by Fable and Sol. Asked for the single biggest
source of wasted rounds, Sol answered: 'The builder hands work to the
reviewer before trying to falsify it adversarially.' Round 2's prompt had
asked the reviewer to check publication races and pid reuse that the builder
had not tested first — using review as the builder's test phase.

Step 1 is now that discipline, and both reviewers shaped it.

Sol rejected the first draft as unexecutable exhortation and as a
contradiction of this file's own scoped-TDD rule. It ships as a checklist
with a done condition — one row per claim and per named risk, recording
claim/risk | falsifier | route | evidence, routed through the existing
EVAL / plain-assert / JUDGMENT-ONLY three-way call, with separate
known-success and known-failure rows for every detector.

Sol also caught the draft breaking this repo's claim rule, which is the
sharper correction: it asserted 'each one cost an architectural round' and
'could have falsified in minutes' — neither measured, neither timed — and
stated a causal collapse where the record supports only a sequence. The
Why now says what the record shows.

The direction-demand folds into the run-reviewer step: a prompt must ask
for the smallest change that would satisfy the certify condition, and that
direction is a proposal under the dialogue loop, not a command. The first
draft said 'implement verbatim', contradicting the loop's explicit right to
dispute a finding with evidence.

Fable caught the ordering: falsify-first was written after the step that
launches the leg, so a reader executing top-to-bottom had already sent the
review before reaching the rule governing what comes first.
BaseInfinity added a commit that referenced this pull request Aug 14, 2026
Post-merge audit of PR #595 by Fable and Sol. Asked for the single biggest
source of wasted rounds, Sol answered: 'The builder hands work to the
reviewer before trying to falsify it adversarially.' Round 2's prompt had
asked the reviewer to check publication races and pid reuse that the builder
had not tested first — using review as the builder's test phase.

Step 1 is now that discipline, and both reviewers shaped it.

Sol rejected the first draft as unexecutable exhortation and as a
contradiction of this file's own scoped-TDD rule. It ships as a checklist
with a done condition — one row per claim and per named risk, recording
claim/risk | falsifier | route | evidence, routed through the existing
EVAL / plain-assert / JUDGMENT-ONLY three-way call, with separate
known-success and known-failure rows for every detector.

Sol also caught the draft breaking this repo's claim rule, which is the
sharper correction: it asserted 'each one cost an architectural round' and
'could have falsified in minutes' — neither measured, neither timed — and
stated a causal collapse where the record supports only a sequence. The
Why now says what the record shows.

The direction-demand folds into the run-reviewer step: a prompt must ask
for the smallest change that would satisfy the certify condition, and that
direction is a proposal under the dialogue loop, not a command. The first
draft said 'implement verbatim', contradicting the loop's explicit right to
dispute a finding with evidence.

Fable caught the ordering: falsify-first was written after the step that
launches the leg, so a reader executing top-to-bottom had already sent the
review before reaching the rule governing what comes first.
A codex review leg handed a pipe nobody closes blocks reading stdin before it
ever contacts the model. On 2026-08-13 two legs were waited on for 51 and 28
minutes, with the loop's "still working, or done?" decision made twice
against a process that had never started.

Two attempts to DETECT that from outside were falsified by Sol (GPT-5.6 high)
with running code.

Watching the output file and the process table: `-o` output carries no
`tokens used` block, so a successful review read as dead; `tokens used`
appears in codex's echoed prompt, so a crashed leg read as complete; an fd
reported as PIPE may be at EOF and healthy; the hang's byte signature is
machine-specific (39 here, 143 elsewhere); and enumerating holders of the
file counted readers, so a `tail -f` flipped the verdict.

Recording the child's pid and exit status in sidecars for a waiter to read:
the child exits before the status is published, so a successful leg read as
dead; killing only the launcher left a status that could never arrive while
the child pid still looked alive; a stale status was read as this run's; and
a reused pid was indistinguishable from the original process.

Both are the same mistake — a second observer reconstructing what the first
process already knows. So there is one script and no observer. The launcher
gives the child `< /dev/null`, whatever it inherited itself, and execs codex
so the caller sees codex's own exit status with nothing in between that could
die separately. Run as a background task, that status IS the verdict.

The sidecars are gone with the waiter: with no reader they were only a way to
be stale.

Verified against real codex 0.147.0 with the launcher's own stdin on an
unclosed fifo — the shape that reproduces a live hang — the leg completed,
exit 0. This commit's own review legs were launched through it.

skills/sdlc/SKILL.md (+ cowork copy): steps 1 and 2 no longer contradict each
other — the raw `codex exec` line is now the inside of the launcher rather
than a parallel instruction — and the falsified 39-byte claim is gone.

9 tests. The elaborate fixtures went with the design that needed them.
#590)

Sol round 3, both measured against codex 0.147.0:

The text claimed the launcher passes '-o <output>'. It does not, and doing it
is actively wrong — pointing '-o' and the redirect at one path makes codex
truncate most of the transcript and duplicate the final response.

Reusing an output path for a relaunch truncates it while the old leg is still
alive and still appending, so two transcripts interleave and both legs report
success. Cancel and await the owned task before reusing the path, or relaunch
to a unique one.
BaseInfinity added a commit that referenced this pull request Aug 15, 2026
…tly not happen (#620)

* release: v1.98.0 — the review loop terminates, and a leg cannot silently not happen

Nine consumer-affecting commits have been sitting on main since the v1.97.0
release commit, reaching nobody. release-drift.yml filed #618 for exactly this,
and the v1.98.0 milestone has been closed with eleven issues and no tag.

What consumers get that they did not have:

  #606  the review loop has a stated termination condition
  #590  a launcher for review legs, and a gate that requires it
        (#595 the owner-not-observer design, #596 falsify-before-review,
         #610 the gate lane that refuses a hand-typed leg)
  #613  the build is an input to a review verdict, not background noise
  #588  the commit gate stops matching `git commit` in prose
  #581  a JSON-escaped newline no longer hides an invocation from the gate
  #533  the review gate no longer makes the review protocol uncommittable
  #577  the review contract as one prose batch

The two hook fixes are the ones worth reading twice: both REFUSED legitimate
work on consumer machines. A commit whose message merely mentioned the phrase
was blocked, and a command split across an escaped newline walked straight
past. Consumers have been running both since v1.97.0.

package.json and CHANGELOG.md only. Every item was certified on its own PR;
this commit ships them.

doc-consistency 137/0. release-workflow 14/0. release-drift 21/0.

* fix(changelog): restore the 1.97.0 header this release deleted

Review found it, and it was the worst kind of mistake to make in a release PR:
the edit that inserted the 1.98.0 entry anchored on the `## [1.97.0]` header
and replaced it, so the header vanished. Headers ran 1.98.0 -> 1.96.0, and the
whole v1.97.0 entry — the Cowork classifier removal and everything beneath it —
read as part of THIS release.

A changelog that attributes a previous release's work to the current one is
worse than a missing entry: it is confidently wrong, and it ships to every
consumer as the record of what they just installed.

Headers now read 1.98.0 (7), 1.97.0 (65), 1.96.0 (153), 1.95.0 (228).

No suite checks that changelog version headers are present and ordered, which
is why CI passed on it. Filed separately rather than fixed here — this PR's own
review bar blocks on any file other than package.json and CHANGELOG.md
changing, and a release PR is the wrong place to grow test surface.

The same review verified every load-bearing claim in the new entry against its
certified source and found each accurate verbatim, and confirmed no
consumer-affecting change is missing from it.

* fix(release): bump every version stamp, and correct two false CHANGELOG claims

CI validate was red on 3e5d36a: test-self-update's version-parity check,
twice. SDLC.md:123 enumerates nine locations the version string lives in;
this release had bumped package.json and CHANGELOG.md only. Bumped the rest.

The strings left at 1.97.0 are deliberate — transcripts of observed command
output, and historical "removed in v1.97.0" references. Bumping those would
falsify them.

Two CHANGELOG claims were wrong and are corrected:

- It said consumers receive scripts/run-review-leg.sh. They do not.
  `npm pack --dry-run | grep -c scripts/` is 0; package.json's files list
  has no scripts/ entry. The #610 gate lane ships in hooks/; the launcher it
  names is #594, still open. A "Not in this release" section now says so.

- It said all three commit-gate defects blocked legitimate work. #588 and
  #533 did; #581 failed OPEN. Opposite failure directions, and reading one
  for the other gives a consumer the wrong idea of what the gate is for.

The review bar itself was also wrong. Rule 3 admitted only package.json and
CHANGELOG.md, which is mutually exclusive with CI's parity test — a release
PR satisfying it cannot go green. Amended to permit the SDLC.md:123 stamps.

* docs(sdlc): add the Cowork marketplace manifest to the version-bump checklist

Both reviewers found this independently on the v1.98.0 release PR — Sol as a
P1, Fable as a P3. SDLC.md:123 enumerates where the version string lives, and
the amended release-PR rule points at that enumeration. It omitted
cowork/.claude-plugin/marketplace.json, which every release must bump:
test-cowork-drift.sh:89 enforces marketplace/plugin parity and :283 enforces
Cowork/package parity, so moving package.json forces both Cowork manifests.

So a release that satisfies the checklist still fails CI, and a release that
passes CI is outside the checklist. Same shape as the rule-3 defect this PR
already fixed, one level down.

The bump was correct; the checklist was stale. Fixed the checklist.

Also: ROADMAP.md's "Last release: vX.Y.Z" marker joins the list — this release
had to bump it and it was not named — and the "no dedicated test yet" clause
now excludes the Cowork marketplace, which test-cowork-drift does cover.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

codex exec hangs forever on stdin; every cross-model review leg needs < /dev/null

1 participant