Repository navigation
Reviewing a contract before it leaves the machine is now the default - #41
Merged
Merged
Conversation
The critic caught two real defects in one session — a "guaranteed null" that nothing enforced, and an experiment record asserting a property a reader had to go and verify. Both were written minutes after landing a Layer 2 control against that exact mistake. Running it is currently something someone has to remember, which makes it Layer 3. This is the Layer 2 version. On `git push` and `gh pr create`, the measurement contracts changed on the branch — benchmark/rubrics/*.yaml, templates/*.yaml, experiments/*.md — go to tools/opencode-review.sh and the findings land in findings/opencode/. Push is the trigger because it is the moment the artifact stops being yours alone. Command filtering is `"if": "Bash(git push:*)"` and backgrounding is `"async": true`, both from the settings schema rather than hand-rolled in the script — the hook is not spawned at all for a non-matching command, and the push has already completed before the reviewer starts. The script re-checks the command anyway, because its test runs it directly. It exits 0 on every path: missing opencode, a broken reviewer, malformed stdin, no git, no merge-base. A reviewer that can break `git push` gets removed within a day, and then nothing is reviewed at all. opencode-review.test.sh covers 16 cases with gh, git, opencode and the reviewer all stubbed — no network, no tokens, no model calls — and runs in CI. Half the cases assert the reviewer was NOT called: a hook that silently does nothing is worse than no hook. One of them only passes because the isolated PATH contains no opencode at all; removing the stub was not enough, since the developer's real binary was still found and the case passed for the wrong reason. Scope is project .claude/settings.json — committed, team-wide. Anyone who clones the repo gets it. LAB_REVIEW_HOOK=0 turns it off. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
hermanngeorge15
added a commit
that referenced
this pull request
Aug 27, 2026
The harness moves to main on its own. #41 merged a hook that calls tools/opencode-review.sh, and that script was only ever on b01-review-harness — so the hook has been inert on main since it landed, silently exiting 0 on its own `[ -x tools/opencode-review.sh ]` guard. Exactly the failure its README warns about. The harness is validated; it is the rubric in #18 that is not, and the harness does not depend on it. line-level lab-critic ollama-cloud/glm-5.2 every section, every finding bound to a concrete failure acceptance lab-acceptance ollama-cloud/minimax-m3 one verdict on the artifact Finding and deciding are different jobs. Running both on one model means one set of blind spots covers both, and until now everything — nine findings files — was deepseek-v4-pro doing both. Nothing after this is comparable to those without a re-run; both models are stamped into the provenance header so a change mid-experiment is provable rather than deniable. The acceptance pass reads the artifact AND the line-level findings, and is deliberately NOT given the recurrence table: how often a finding recurred is a fact about the line-level model's detection threshold, and a gate that weights by it measures that model rather than the artifact. It may put a finding it cannot substantiate into `disputed`, and it did on its first real run. The gate is advisory. REJECT is recorded and printed; the script still exits 0, because the hook that calls it must never fail a push. That is L3 and is labelled L3. LAB_ACCEPT_STRICT=1 is the L2 version — REJECT exits 3. Nothing sets it yet. First real run, against experiments/E-001: REJECT, four blocking findings. One is a contradiction this session introduced — Predictions registers 24 cells, the MDE table still said 20, and the decision rule's threshold flips on which a reader sees first. Provenance for the two choices, because neither was the user's explicit call: the request said deepseek for acceptance and its table said minimax-m3. The table wins here as the more considered artifact, and the gate shape — one verdict on the whole artifact rather than a criteria checklist or a finding-verifier — follows from calling it "the nominated gate". Both are one env var from being reversed. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Committed to the repo, so anyone who clones it gets the hook. Not a personal config.
.claude/settings.json.claude/hooks/opencode-review.shchmod +x.claude/hooks/opencode-review.test.shgh/git/opencode/ the reviewer all stubbed — no network, no tokens.claude/hooks/README.mdWhat it does
On
git pushandgh pr create, the measurement contracts changed on the branch go totools/opencode-review.sh— the independent critic, a different model family — and findings land infindings/opencode/.Three globs, listed at the top of the script:
benchmark/rubrics/*.yaml,templates/*.yaml,experiments/*.md. A phase README changing does not need an adversarial reviewer; a rubric leaving the machine does.Why
In one session the critic caught a "guaranteed null" that nothing enforced, and an experiment record asserting a property a reader had to go and verify by hand. Both were written minutes after landing a Layer 2 control against that exact mistake.
A review you have to remember to run is Layer 3. Push is the right trigger because it is the moment the artifact stops being yours alone.
Nothing hand-rolled that the schema provides
{ "type": "command", "command": ".claude/hooks/opencode-review.sh", "if": "Bash(git push:*)", "async": true, "timeout": 900 }ifuses permission-rule syntax, so the hook is not spawned at all for a non-matching command — nocasestatement filtering inside the script.async: truekeeps it off the critical path, so the push has already completed before the reviewer starts — no( … ) &self-detach. The script still re-checks the command, because its test runs it directly.It cannot break a push
Exits 0 on every path: missing
opencode, a reviewer that exits 1, malformed stdin, no git, no merge-base. A reviewer that can breakgit pushgets removed within a day, and then nothing is reviewed at all.The test
Half the cases assert the reviewer was NOT called. A hook that silently does nothing is worse than no hook, and that is the failure mode this repo keeps meeting —
--dirsilently selecting the default agent and exiting 0, a suite printing "all 6 cases" while eleven ran.One case only passes because of a deliberately minimal
PATHcontaining noopencode. Removing the stub was not enough: the developer's real binary was still on$PATH,command -vfound it, and the case passed for the wrong reason. Caught while writing the test, which is the argument for writing it.Runs in CI, along with shellcheck over
.claude/hooks.Scope
Project
.claude/settings.json— committed, team-wide.LAB_REVIEW_HOOK=0turns it off for one command;LAB_REVIEW_RUNSoverrides-n(default 2, because a single run at temperature 0 is a lower bound).After merging, open
/hooksonce or restart — the settings watcher only watches directories that had a settings file when the session started, so a brand-new.claude/settings.jsonis not picked up mid-session.🤖 Generated with Claude Code