Skip to content

Reviewing a contract before it leaves the machine is now the default - #41

Merged
hermanngeorge15 merged 1 commit into
mainfrom
chore/opencode-review-hook
Aug 27, 2026
Merged

hermanngeorge15 merged 1 commit into
mainfrom
chore/opencode-review-hook

Conversation

@hermanngeorge15

Copy link
Copy Markdown
Contributor

Committed to the repo, so anyone who clones it gets the hook. Not a personal config.

File Role
.claude/settings.json the wiring
.claude/hooks/opencode-review.sh the script, chmod +x
.claude/hooks/opencode-review.test.sh 16 cases, gh / git / opencode / the reviewer all stubbed — no network, no tokens
.claude/hooks/README.md why, and how to install it elsewhere

What it does

On git push and gh pr create, the measurement contracts changed on the branch go to tools/opencode-review.sh — the independent critic, a different model family — and findings land in findings/opencode/.

Three globs, listed at the top of the script: benchmark/rubrics/*.yaml, templates/*.yaml, experiments/*.md. A phase README changing does not need an adversarial reviewer; a rubric leaving the machine does.

Why

In one session the critic caught a "guaranteed null" that nothing enforced, and an experiment record asserting a property a reader had to go and verify by hand. Both were written minutes after landing a Layer 2 control against that exact mistake.

A review you have to remember to run is Layer 3. Push is the right trigger because it is the moment the artifact stops being yours alone.

Nothing hand-rolled that the schema provides

{ "type": "command", "command": ".claude/hooks/opencode-review.sh",
  "if": "Bash(git push:*)", "async": true, "timeout": 900 }

if uses permission-rule syntax, so the hook is not spawned at all for a non-matching command — no case statement filtering inside the script. async: true keeps it off the critical path, so the push has already completed before the reviewer starts — no ( … ) & self-detach. The script still re-checks the command, because its test runs it directly.

It cannot break a push

Exits 0 on every path: missing opencode, a reviewer that exits 1, malformed stdin, no git, no merge-base. A reviewer that can break git push gets removed within a day, and then nothing is reviewed at all.

The test

ok    push, no changes                             exit 0, 0 reviewer call(s)
ok    git status is not a push                     exit 0, 0 reviewer call(s)
ok    git pushd is not a push                      exit 0, 0 reviewer call(s)
ok    gh pr view is not create                     exit 0, 0 reviewer call(s)
ok    push with a changed rubric                   exit 0, 1 reviewer call(s)
ok    reviewer argv                                argv carries the artifact
ok    README change is not reviewable              exit 0, 0 reviewer call(s)
ok    reviewer exits 1                             exit 0, 1 reviewer call(s)
ok    opencode not installed                       exit 0, 0 reviewer call(s)
ok    empty stdin / not JSON / no command / wrong shape
opencode-review.test: all 16 cases behaved as specified.

Half the cases assert the reviewer was NOT called. A hook that silently does nothing is worse than no hook, and that is the failure mode this repo keeps meeting — --dir silently selecting the default agent and exiting 0, a suite printing "all 6 cases" while eleven ran.

One case only passes because of a deliberately minimal PATH containing no opencode. Removing the stub was not enough: the developer's real binary was still on $PATH, command -v found it, and the case passed for the wrong reason. Caught while writing the test, which is the argument for writing it.

Runs in CI, along with shellcheck over .claude/hooks.

Scope

Project .claude/settings.json — committed, team-wide. LAB_REVIEW_HOOK=0 turns it off for one command; LAB_REVIEW_RUNS overrides -n (default 2, because a single run at temperature 0 is a lower bound).

After merging, open /hooks once or restart — the settings watcher only watches directories that had a settings file when the session started, so a brand-new .claude/settings.json is not picked up mid-session.

🤖 Generated with Claude Code

The critic caught two real defects in one session — a "guaranteed null" that
nothing enforced, and an experiment record asserting a property a reader had to go
and verify. Both were written minutes after landing a Layer 2 control against that
exact mistake. Running it is currently something someone has to remember, which
makes it Layer 3. This is the Layer 2 version.

On `git push` and `gh pr create`, the measurement contracts changed on the branch —
benchmark/rubrics/*.yaml, templates/*.yaml, experiments/*.md — go to
tools/opencode-review.sh and the findings land in findings/opencode/. Push is the
trigger because it is the moment the artifact stops being yours alone.

Command filtering is `"if": "Bash(git push:*)"` and backgrounding is `"async": true`,
both from the settings schema rather than hand-rolled in the script — the hook is not
spawned at all for a non-matching command, and the push has already completed before
the reviewer starts. The script re-checks the command anyway, because its test runs it
directly.

It exits 0 on every path: missing opencode, a broken reviewer, malformed stdin, no
git, no merge-base. A reviewer that can break `git push` gets removed within a day,
and then nothing is reviewed at all.

opencode-review.test.sh covers 16 cases with gh, git, opencode and the reviewer all
stubbed — no network, no tokens, no model calls — and runs in CI. Half the cases assert
the reviewer was NOT called: a hook that silently does nothing is worse than no hook.
One of them only passes because the isolated PATH contains no opencode at all; removing
the stub was not enough, since the developer's real binary was still found and the case
passed for the wrong reason.

Scope is project .claude/settings.json — committed, team-wide. Anyone who clones the
repo gets it. LAB_REVIEW_HOOK=0 turns it off.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@hermanngeorge15
hermanngeorge15 merged commit ee0d49a into main Aug 27, 2026
4 checks passed
@hermanngeorge15
hermanngeorge15 deleted the chore/opencode-review-hook branch August 27, 2026 10:41
hermanngeorge15 added a commit that referenced this pull request Aug 27, 2026
The harness moves to main on its own. #41 merged a hook that calls
tools/opencode-review.sh, and that script was only ever on b01-review-harness — so
the hook has been inert on main since it landed, silently exiting 0 on its own
`[ -x tools/opencode-review.sh ]` guard. Exactly the failure its README warns about.
The harness is validated; it is the rubric in #18 that is not, and the harness does
not depend on it.

  line-level   lab-critic      ollama-cloud/glm-5.2     every section, every finding
                                                        bound to a concrete failure
  acceptance   lab-acceptance  ollama-cloud/minimax-m3  one verdict on the artifact

Finding and deciding are different jobs. Running both on one model means one set of
blind spots covers both, and until now everything — nine findings files — was
deepseek-v4-pro doing both. Nothing after this is comparable to those without a re-run;
both models are stamped into the provenance header so a change mid-experiment is
provable rather than deniable.

The acceptance pass reads the artifact AND the line-level findings, and is deliberately
NOT given the recurrence table: how often a finding recurred is a fact about the
line-level model's detection threshold, and a gate that weights by it measures that
model rather than the artifact. It may put a finding it cannot substantiate into
`disputed`, and it did on its first real run.

The gate is advisory. REJECT is recorded and printed; the script still exits 0, because
the hook that calls it must never fail a push. That is L3 and is labelled L3.
LAB_ACCEPT_STRICT=1 is the L2 version — REJECT exits 3. Nothing sets it yet.

First real run, against experiments/E-001: REJECT, four blocking findings. One is a
contradiction this session introduced — Predictions registers 24 cells, the MDE table
still said 20, and the decision rule's threshold flips on which a reader sees first.

Provenance for the two choices, because neither was the user's explicit call: the
request said deepseek for acceptance and its table said minimax-m3. The table wins here
as the more considered artifact, and the gate shape — one verdict on the whole artifact
rather than a criteria checklist or a finding-verifier — follows from calling it "the
nominated gate". Both are one env var from being reversed.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant