pr-referee

We built pr-referee, a code reviewer for pull requests that only reports what it can prove. It runs the old and the new version of every changed function on the same inputs, and a finding is reported only when it comes with the input that shows it. It is for teams that want a review check strong enough to block a merge: one that raises no false alarms, says when it could not check something, and publishes how often that happens. It also scores itself against an LLM reviewer (qwen2.5-coder:14b) on the same labelled diffs.

What it does

• Extracts both versions of every changed module-level function from two git revisions.
• Builds argument sets from the repo's own tests, from type hints and from known corner cases.
• Runs both versions on each argument set and compares return values and exceptions.
• Reports five kinds of fact, each with its evidence, and nothing else:

kind	what it means	the evidence
behaviour-change	old and new return different things	the argument set, and both results
swallowed-error	an exception path that used to propagate no longer does	the handler, quoted
widened-catch	an except clause now catches strictly more	before and after breadth
removed-branch	a conditional test present before is gone	the if test, quoted
weakened-tests	the function became harder to tell from a broken version of itself	mutants distinguished, before and after

• When nothing is found it says "nothing here could be proven wrong", not "looks good".
• Counts and reports every function it could not execute (missing dependency, an object it cannot build), instead of skipping it silently.
• --fail-on-finding turns it into a CI gate.
• bench builds labelled diffs from a real repository and scores the referee and the model side by side.

Sample input / output

pr-referee review on a change that narrows toolz's get_in to a bare except Exception: return default:

  [behaviour-change] toolz/dicttoolz.py::get_in
    this change alters what the function returns
    | get_in([1, 2, 3], [1, 2, 3], 'b', True)
    |   before: raise: TypeError: 'int' object is not subscriptable
    |   after:  ok: 'b'
    -> 24 of 30 inputs disagree

  [widened-catch] toolz/dicttoolz.py::get_in
    the exception clause widened to `Exception`
    | broadest handler before: level 1; after: level 2
    -> A broader clause also catches the errors nobody meant to handle.

  [removed-branch] toolz/dicttoolz.py::get_in
    1 conditional path(s) no longer present
    | `if no_default`
    -> Whatever the removed branch did is now unreachable.

python demo.py (no model, about 5 seconds; toolz is vendored under targets/):

pr-referee: can it tell a behaviour-changing diff from an equivalent one?

building labelled diffs from toolz...
  6 cases: 1 provably behaviour-changing, 5 equivalent
running the referee...

==============================================================================
REVIEWER COMPARISON - 6 labelled diffs from toolz
==============================================================================
  reviewer           TP   FP   TN   FN  abst  precision   recall      F1
  --------------------------------------------------------------------------
  pr-referee          1    0    2    0     3     100.0%   100.0%  100.0%

  the referee also raised 2 structural finding(s) - swallowed
  errors, widened catches, removed branches - which are not behaviour claims
  and are therefore not scored above.

  took 5s

Results

116 labelled diffs from toolz: 24 proven to change behaviour (single-point mutations kept only when an independent 240-input pool shows a difference), 92 equivalent by construction (rename a local, extract a return temporary, x = x+1 to x += 1).

reviewer	TP	FP	TN	FN	abstained	precision	recall	F1
pr-referee	22	0	45	0	49	100.0%	100.0%	100.0%
qwen2.5-coder:14b	18	1	91	6	0	94.7%	75.0%	83.7%

• The referee raised no false alarm and decided 67 of 116 diffs (57.8%); it abstained on the rest.
• The model decided all 116, missed 6 of 24 real behaviour changes and raised one false alarm.
• Shown both the before and the after, the model reached 94.7% precision, against 44.8% when code-llm-lab showed it one version and asked "is there a bug?".
• The referee also raised 35 structural findings (swallowed errors, widened catches, removed branches), which are not scored above.

Full run: docs/RESULTS.md and docs/bench-result.json. How the labels are built: docs/METHOD.md.

Tech stack

• Python 3.11+, zero runtime dependencies
• ast for function extraction and the structural checks, subprocess for running both versions in isolation, git for the two revisions
• urllib to call a local Ollama server (qwen2.5-coder:14b) for the comparison arm of bench only
• pytest, ruff, hatchling, GitHub Actions

Run it

git clone https://github.com/hammasbuilds/pr-referee
cd pr-referee
uv sync --all-groups

uv run python demo.py

# review a branch
uv run pr-referee review /path/to/repo --base main --head HEAD

# in CI, fail the build on a provable finding
uv run pr-referee review . --base origin/main --head HEAD --fail-on-finding

# score the referee against the model on labelled diffs (--no-llm skips the model)
uv run pr-referee bench /path/to/repo --limit 140

• review needs no model. Only the model arm of bench needs Ollama with qwen2.5-coder:14b.
• It reviews module-level functions; methods are not reviewed, because building an instance means running the code under review.
• The target repo's own dependencies must be importable, or its functions are counted as "could not execute".

Code layout: src/pr_referee/ holds extract.py (both versions of each function), values.py (argument sets), engine/differential.py and engine/mutation.py, checks/statics.py, referee.py, reviewer.py (the model arm), bench.py and score.py.

Related: repo-surgeon uses the same engine to propose changes and refuse the ones it cannot prove.

Licence

MIT, see LICENSE. targets/toolz is a vendored copy of toolz (BSD-3-Clause).