fix(review): give review legs an owner instead of an observer (#590) - #595
Conversation
|
CROSS-MODEL-CLEARANCE Sol (GPT-5.6 Sol, high) — round 4 of 4. Findings by round: 5 P1s, 4 P1s, 1 P1, 0. Verbatim: "1. Yes. The diff exactly satisfies the named certify condition. 2. VERDICT: CERTIFIED — 10/10." {
"confidence": 99,
"reviewer": "gpt-5.6-sol-high",
"sha": "9f67c9b9d6930dc40b46a85ad5a09add2cd38260",
"verdict": "YES"
} |
|
CROSS-MODEL-CLEARANCE Fable (high) — design authority across all four rounds. Ruled twice against patching: round 1 that observing an already-running leg is intrinsically unsound, round 2 that the reviewer's smallest-change prescription should be taken verbatim and the sidecars dropped with the waiter. Grounds for clearance: the final design eliminates the failure class rather than detecting it; live verification against real codex in the exact hang shape; this PR's own review legs ran through the launcher; all gate-relevant suites pass; the one open item (#594) is filed, quoted, and correctly non-blocking. {
"confidence": 97,
"reviewer": "fable-high",
"sha": "9f67c9b9d6930dc40b46a85ad5a09add2cd38260",
"verdict": "YES"
} |
test-workflow-triggers.sh requires CONTRIBUTING.md to name every script CI validate runs. The new step was added to ci.yml without it, so CI failed on its own consistency guard.
|
CROSS-MODEL-CLEARANCE Sol (GPT-5.6 Sol, high) — round 5, re-bound to the new head after the CONTRIBUTING.md CI fix. Verbatim: "No—the delta does not affect the verdict. VERDICT: CERTIFIED". Supersedes the clearance bound to 9f67c9b. CI validate green (run 31756006876, headSha 8c8f49a). {
"confidence": 100,
"reviewer": "gpt-5.6-sol-high",
"sha": "8c8f49a337832d9ef2e0f5e47ef03e5651bc81d0",
"verdict": "YES"
} |
|
CROSS-MODEL-CLEARANCE Fable (high) — re-bound to the new head. The delta since the prior clearance is one line in CONTRIBUTING.md, added because the repo's own consistency guard requires every CI-run script to be listed there. It changes no behaviour and the guard that demanded it now passes. Prior grounds stand. {
"confidence": 97,
"reviewer": "fable-high",
"sha": "8c8f49a337832d9ef2e0f5e47ef03e5651bc81d0",
"verdict": "YES"
} |
|
DUAL CROSS-MODEL CERTIFIED MERGE Cleared by: gpt-5.6-sol-high (posted by @BaseInfinity), fable-high (posted by @BaseInfinity) — both bound to Merge-evidence path(s) this authorised:
Verified: CI ATTESTED, NOT AUTHENTICATED. Both clearances were posted by the same Posted by |
Post-merge audit of PR #595 by Fable and Sol. Asked for the single biggest source of wasted rounds, Sol answered: 'The builder hands work to the reviewer before trying to falsify it adversarially.' Round 2's prompt had asked the reviewer to check publication races and pid reuse that the builder had not tested first — using review as the builder's test phase. Step 1 is now that discipline, and both reviewers shaped it. Sol rejected the first draft as unexecutable exhortation and as a contradiction of this file's own scoped-TDD rule. It ships as a checklist with a done condition — one row per claim and per named risk, recording claim/risk | falsifier | route | evidence, routed through the existing EVAL / plain-assert / JUDGMENT-ONLY three-way call, with separate known-success and known-failure rows for every detector. Sol also caught the draft breaking this repo's claim rule, which is the sharper correction: it asserted 'each one cost an architectural round' and 'could have falsified in minutes' — neither measured, neither timed — and stated a causal collapse where the record supports only a sequence. The Why now says what the record shows. The direction-demand folds into the run-reviewer step: a prompt must ask for the smallest change that would satisfy the certify condition, and that direction is a proposal under the dialogue loop, not a command. The first draft said 'implement verbatim', contradicting the loop's explicit right to dispute a finding with evidence. Fable caught the ordering: falsify-first was written after the step that launches the leg, so a reader executing top-to-bottom had already sent the review before reaching the rule governing what comes first.
Post-merge audit of PR #595 by Fable and Sol. Asked for the single biggest source of wasted rounds, Sol answered: 'The builder hands work to the reviewer before trying to falsify it adversarially.' Round 2's prompt had asked the reviewer to check publication races and pid reuse that the builder had not tested first — using review as the builder's test phase. Step 1 is now that discipline, and both reviewers shaped it. Sol rejected the first draft as unexecutable exhortation and as a contradiction of this file's own scoped-TDD rule. It ships as a checklist with a done condition — one row per claim and per named risk, recording claim/risk | falsifier | route | evidence, routed through the existing EVAL / plain-assert / JUDGMENT-ONLY three-way call, with separate known-success and known-failure rows for every detector. Sol also caught the draft breaking this repo's claim rule, which is the sharper correction: it asserted 'each one cost an architectural round' and 'could have falsified in minutes' — neither measured, neither timed — and stated a causal collapse where the record supports only a sequence. The Why now says what the record shows. The direction-demand folds into the run-reviewer step: a prompt must ask for the smallest change that would satisfy the certify condition, and that direction is a proposal under the dialogue loop, not a command. The first draft said 'implement verbatim', contradicting the loop's explicit right to dispute a finding with evidence. Fable caught the ordering: falsify-first was written after the step that launches the leg, so a reader executing top-to-bottom had already sent the review before reaching the rule governing what comes first.
A codex review leg handed a pipe nobody closes blocks reading stdin before it ever contacts the model. On 2026-08-13 two legs were waited on for 51 and 28 minutes, with the loop's "still working, or done?" decision made twice against a process that had never started. Two attempts to DETECT that from outside were falsified by Sol (GPT-5.6 high) with running code. Watching the output file and the process table: `-o` output carries no `tokens used` block, so a successful review read as dead; `tokens used` appears in codex's echoed prompt, so a crashed leg read as complete; an fd reported as PIPE may be at EOF and healthy; the hang's byte signature is machine-specific (39 here, 143 elsewhere); and enumerating holders of the file counted readers, so a `tail -f` flipped the verdict. Recording the child's pid and exit status in sidecars for a waiter to read: the child exits before the status is published, so a successful leg read as dead; killing only the launcher left a status that could never arrive while the child pid still looked alive; a stale status was read as this run's; and a reused pid was indistinguishable from the original process. Both are the same mistake — a second observer reconstructing what the first process already knows. So there is one script and no observer. The launcher gives the child `< /dev/null`, whatever it inherited itself, and execs codex so the caller sees codex's own exit status with nothing in between that could die separately. Run as a background task, that status IS the verdict. The sidecars are gone with the waiter: with no reader they were only a way to be stale. Verified against real codex 0.147.0 with the launcher's own stdin on an unclosed fifo — the shape that reproduces a live hang — the leg completed, exit 0. This commit's own review legs were launched through it. skills/sdlc/SKILL.md (+ cowork copy): steps 1 and 2 no longer contradict each other — the raw `codex exec` line is now the inside of the launcher rather than a parallel instruction — and the falsified 39-byte claim is gone. 9 tests. The elaborate fixtures went with the design that needed them.
#590) Sol round 3, both measured against codex 0.147.0: The text claimed the launcher passes '-o <output>'. It does not, and doing it is actively wrong — pointing '-o' and the redirect at one path makes codex truncate most of the transcript and duplicate the final response. Reusing an output path for a relaunch truncates it while the old leg is still alive and still appending, so two transcripts interleave and both legs report success. Cancel and await the owned task before reusing the path, or relaunch to a unique one.
…tly not happen (#620) * release: v1.98.0 — the review loop terminates, and a leg cannot silently not happen Nine consumer-affecting commits have been sitting on main since the v1.97.0 release commit, reaching nobody. release-drift.yml filed #618 for exactly this, and the v1.98.0 milestone has been closed with eleven issues and no tag. What consumers get that they did not have: #606 the review loop has a stated termination condition #590 a launcher for review legs, and a gate that requires it (#595 the owner-not-observer design, #596 falsify-before-review, #610 the gate lane that refuses a hand-typed leg) #613 the build is an input to a review verdict, not background noise #588 the commit gate stops matching `git commit` in prose #581 a JSON-escaped newline no longer hides an invocation from the gate #533 the review gate no longer makes the review protocol uncommittable #577 the review contract as one prose batch The two hook fixes are the ones worth reading twice: both REFUSED legitimate work on consumer machines. A commit whose message merely mentioned the phrase was blocked, and a command split across an escaped newline walked straight past. Consumers have been running both since v1.97.0. package.json and CHANGELOG.md only. Every item was certified on its own PR; this commit ships them. doc-consistency 137/0. release-workflow 14/0. release-drift 21/0. * fix(changelog): restore the 1.97.0 header this release deleted Review found it, and it was the worst kind of mistake to make in a release PR: the edit that inserted the 1.98.0 entry anchored on the `## [1.97.0]` header and replaced it, so the header vanished. Headers ran 1.98.0 -> 1.96.0, and the whole v1.97.0 entry — the Cowork classifier removal and everything beneath it — read as part of THIS release. A changelog that attributes a previous release's work to the current one is worse than a missing entry: it is confidently wrong, and it ships to every consumer as the record of what they just installed. Headers now read 1.98.0 (7), 1.97.0 (65), 1.96.0 (153), 1.95.0 (228). No suite checks that changelog version headers are present and ordered, which is why CI passed on it. Filed separately rather than fixed here — this PR's own review bar blocks on any file other than package.json and CHANGELOG.md changing, and a release PR is the wrong place to grow test surface. The same review verified every load-bearing claim in the new entry against its certified source and found each accurate verbatim, and confirmed no consumer-affecting change is missing from it. * fix(release): bump every version stamp, and correct two false CHANGELOG claims CI validate was red on 3e5d36a: test-self-update's version-parity check, twice. SDLC.md:123 enumerates nine locations the version string lives in; this release had bumped package.json and CHANGELOG.md only. Bumped the rest. The strings left at 1.97.0 are deliberate — transcripts of observed command output, and historical "removed in v1.97.0" references. Bumping those would falsify them. Two CHANGELOG claims were wrong and are corrected: - It said consumers receive scripts/run-review-leg.sh. They do not. `npm pack --dry-run | grep -c scripts/` is 0; package.json's files list has no scripts/ entry. The #610 gate lane ships in hooks/; the launcher it names is #594, still open. A "Not in this release" section now says so. - It said all three commit-gate defects blocked legitimate work. #588 and #533 did; #581 failed OPEN. Opposite failure directions, and reading one for the other gives a consumer the wrong idea of what the gate is for. The review bar itself was also wrong. Rule 3 admitted only package.json and CHANGELOG.md, which is mutually exclusive with CI's parity test — a release PR satisfying it cannot go green. Amended to permit the SDLC.md:123 stamps. * docs(sdlc): add the Cowork marketplace manifest to the version-bump checklist Both reviewers found this independently on the v1.98.0 release PR — Sol as a P1, Fable as a P3. SDLC.md:123 enumerates where the version string lives, and the amended release-PR rule points at that enumeration. It omitted cowork/.claude-plugin/marketplace.json, which every release must bump: test-cowork-drift.sh:89 enforces marketplace/plugin parity and :283 enforces Cowork/package parity, so moving package.json forces both Cowork manifests. So a release that satisfies the checklist still fails CI, and a release that passes CI is outside the checklist. Same shape as the rule-3 defect this PR already fixed, one level down. The bump was correct; the checklist was stale. Fixed the checklist. Also: ROADMAP.md's "Last release: vX.Y.Z" marker joins the list — this release had to bump it and it was not named — and the "no dedicated test yet" clause now excludes the Cowork marketplace, which test-cowork-drift does cover.
Closes #590.
The failure
codex execreads stdin to EOF and appends it to the argv prompt — argv and stdin are concatenated, not alternatives. Handed a pipe nobody closes, it blocks on that read before contacting the model, forever.On 2026-08-13 two review legs were waited on for 51 and 28 minutes. The loop's "still working, or done?" decision was made twice against a process that had never started. #341 added
< /dev/nullto the documented invocations in May and the hang still happened, because the leg that hung was typed ad hoc.Two attempts to detect it, both falsified by running code
Sol (GPT-5.6 high) did not argue with either design — it broke them.
Attempt 1, watch the output file and the process table:
tokens usedmarker required-owrites only the last agent message and carries no such block → a successful review reported deadtokens usedmarker presentPIPEtail -fflipped the verdictAttempt 2, record the child's pid and exit status in sidecars for a waiter to read: the child exits before the status is published, so a successful leg read as dead; killing only the launcher left a status that could never arrive while the child pid still looked alive; a stale status was read as this run's; a reused pid was indistinguishable from the original.
Both are the same mistake in different clothes: a second observer reconstructing what the first process already knows.
What shipped
One script, no observer.
scripts/run-review-leg.shgives the child< /dev/null— whatever it inherited itself — andexecs codex, so the caller sees codex's own exit status with nothing in between that could die separately. Run as a background task, that status is the verdict.The sidecars went with the waiter: with no reader they were only a way to be stale.
Verification
codexon PATH that reads stdin to EOF exactly as the real one does — so any test that finishes at all is proof the launcher supplied that EOF. No fixture pretends to be a blocked read; the previous suite passed 5/5 against fixtures that were wrong about the very thing they existed to catch.Review record
Four rounds, findings tapering 5 P1s → 4 P1s → 1 P1 → 0. Final: CERTIFIED 10/10, confidence 99.
Rounds 2 and 3 each asked the reviewer, on a sub-7 score, to name the smallest change that would make it pass. Both times it did, and both times that prescription — not the defect list — is what got implemented. Round 3's fix was two sentences and no code.
Known obligation, filed not closed
Consumers do not receive the launcher:
scripts/does not ship, confirmed vianpm pack --dry-run. The shipped skill therefore describes a mechanism it does not supply — the #573 class. Filed as #594 with the reviewer's ship decision quoted verbatim, to land before or with the next release. It was ruled non-blocking for this PR; it is recorded as open, not resolved.