code-llm-lab

We built twenty measurements of a local coder model (qwen2.5-coder, 3B and 14B, through Ollama) that answer questions a pass@1 score cannot: whether the bigger model is worth its VRAM, when to stop a self-debug loop, whether model-written tests catch anything, and how much of a published score is the prompt rather than the model. It is for engineers deciding how to use a coder model, and for anyone reading MBPP or HumanEval numbers. Every result is executed, not judged: a failing assert is the ground truth. No API key, no hosted call.

Projects

	Project	Tasks	The finding
01	Where a 5x bigger model actually pays	972	Of the 817 tasks either size can solve, the 3B already handles 74.8%, and the 14B loses 36 it gets right (3.7%, stable across splits). On MBPP's held-out 500 the size advantage is +16.2 points against +17.5 on all 972 (see Results).
02	How many self-debug rounds are worth paying for	500	Rounds 1–2 capture 99.3% of everything the loop achieves; rounds 3–5 add 3 tasks for 60% of the compute. Measured with the real traceback as feedback.
03	Do model-written tests catch anything	500	From the implementation, generated suites kill 93.8% of mutants against MBPP's own 85.5% — +8.3 points on 164 suites and 723 mutants, with 3.5× the asserts. From the task description only 16.6% of suites even agree with the reference: a measurement of the spec, not the model.
04	Repair the failure, or throw it away	972	77.0% of first-attempt failures survive both, and the two stay indistinguishable (23 vs 17 discordant, p=0.43) — but they fix nearly disjoint sets, only 4 of 44 overlapping, so running both recovers 23.0%. Giving repair a real traceback instead of the word AssertionError was worth +9 fixes (p=0.035), while rewrite — which never reads one — did not move by a single task.
05	The same task, asked five ways	972	The five phrasings score within 2.1 points of each other — and still disagree about 218 tasks (22.4%), solved under one wording and failed under another. A stable aggregate is not evidence of stable behaviour; the wins and losses cancel.
06	The temperature you benchmark at is not the one you deploy at	200	The ordering flips: T=0 wins pass@1 (77.0%), T=0.7 wins pass@5 (84.5%). Tuning on pass@1 and then sampling five times leaves 7.5 points of achievable pass@5 unclaimed, while T=0.7 costs only 1.4 at k=1.
07	Code → prose → code	972	The roundtrip beats MBPP's own task description by 31.7 points (83.3% vs 51.6%) — the opposite of the expected direction. 333 tasks are solved from the code-derived description and only 25 the other way, a thirteen-to-one exchange. The description was written with the answer in view: a leak, not a spec.
08	Ask a model to review correct code	972	Asked to review MBPP's own reference solutions, it flags 37.3% of them — against 45.5% recall on real bugs. It flags unfamiliar correct code at 1.7× the rate of its own correct code (37.3% vs 21.6%). Precision 34.0%.
09	Ask the model how sure it is	972	It says 100 for 99.0% of tasks, including the 191 it gets wrong — three distinct values across 967 answers, separating right from wrong by 0.1 points. Every threshold from 0 to 80 is identical to not gating, and demanding a perfect 100 still auto-merges 99.5%. The dial is not connected to anything.
10	Bury the task in unrelated examples	972	A 15.7× longer prompt costs only 0.9 points of pass@1 — but 95 tasks (9.8%) change verdict, 52 lost and 43 gained. The aggregate is nearly flat because the churn cancels, not because the irrelevant context is harmless.
11	Refactor without changing behaviour	972	rename breaks 31.8% of solutions and 247 of those 248 are just the function being renamed. idiomatic breaks at the same rate, but 9.2% are genuine silent logic changes — and it is the only refactor that makes code shorter.
12	What it reaches for when nobody asks	12	Three of twelve security tasks fail 5 times out of 5 unprompted - pickle, MD5, path traversal - while SQL injection is handled correctly unasked. Specific lessons, not a posture.
13	Which part of an error message does the work	972	A length-matched padding control scores 7.3%, below the 8.4% no-feedback baseline — so none of the gain is prompt length. Showing the actual value doubles it to 18.3% (p=0.0003).
14	Tell it not to do something	972	Compliance reads 85–100%, and most of it is free: no_recursion scores 99% against a 96% baseline it already met by habit. Only type_hints does real work — 0% → 100%. The already column is the whole finding; without it every constraint looks obeyed.
15	Eight solutions in one response	968	Batching eight costs 4.5 points (79.1% → 74.6%) and 2 of 968 functions go missing. Position matters, but less than it looks: each slot holds different tasks, and against their own single-task baseline every slot degrades except the last.
16	Is a code-tuned model worth it	500	The coder advantage is +11.4pp at 3B and +10.0pp at 14B — it shrinks with scale. And it is two-directional: at 14B the coder wins 69 tasks the generalist loses but drops 19 it solves, so the net gap is 50, not 69.
17	Does writing tests first help	972	Writing tests first costs 5.7 points (80.3% → 74.7%). But any preamble at all costs 2.1, so only 3.6 is attributable to the tests themselves — and carrying them into a fresh context recovers 3.7 of it. The TDD framing does negative work here.
18	Temperature 0 is not the same as deterministic	400	Five identical runs give pass@1 within 0.8 points (stdev 0.29) — while 3 tasks flip between pass and fail with nothing changed at all, and 28 (7.0%) return different text. The score is stable because the flips cancel, which is not the same as the decoding being deterministic.
19	Put an instruction in a code comment	200	Loud injected instructions are obeyed +80.8pp above a 0% control — a hard-coded credential 100% of the time. Rephrased as an ordinary engineering note the same requests still land +26.8pp. Register is worth 54 points of defence, which is not a defence.
20	Add a feature to working code	972	Asked to add input validation, 10.4% of working solutions regress — 80 of those 81 by refusing input the original accepted, not by computing it wrong. Asked to add a log line: 0 of 781. The control arm, which cannot break a caller by construction, still regressed 3 — a 0.4% background rate of gratuitous edits.

Every number above came out of a real run, and each is mirrored in that project's results.json with the model, sample size, temperature and seeds that produced it. The seven longest findings are written out in docs/FINDINGS.md; every project also has its own README.

What it does

• Loads MBPP (974 tasks) and HumanEval (164) from the local Hugging Face cache into one task shape, with MBPP's official test / train / validation splits available.
• Talks to Ollama over plain HTTP with an on-disk generation cache keyed by (model, prompt, temperature, seed), and batches eight requests at a time.
• Executes every candidate in a subprocess with a timeout, keeping pass, fail, error and timeout apart: only fail means the tests actually tested something.
• Stamps every results.json with the configuration that produced it, and the test suite checks the arithmetic each headline rests on (buckets disjoint, pass@k rising with k, drift = gained − lost) and that no committed result has fewer than 50 tasks.

Sample input / output

Project 01 re-run from the generation cache (all 972 MBPP tasks, both model sizes; Ollama pointed at a dead port, zero cache misses), real output:

mbpp: 972 tasks
    pass@1: 781/972  (80.3%)

====================================================================
WHERE THE SIZE GOES - mbpp, 972 tasks
====================================================================
  3B  pass@1 : 62.9%
  14B pass@1 : 80.3%
  difference : +17.5%

  both         575   59.2%  ########################
  big_only     206   21.2%  ########
  small_only    36    3.7%  #
  neither      155   15.9%  ######

  Of the 817 tasks either size can solve, the 3B already handles 74.8%.
  The 14B's entire advantage is 206 tasks (21.2% of the benchmark).
  It also loses 36 tasks the 3B gets right, which bounds how much of the gap is real rather than noise.

Results

The projects table above carries each project's headline. Three that cut across them:

MBPP's training split inflates the score. MBPP's official split puts task_ids 601–974 in training data, public since 2021. On the same cached generations:

split	n	qwen2.5-coder:3b	qwen2.5-coder:14b
test (held out)	500	60.6%	76.8%
train (public since 2021)	372	65.3%	85.2%
gap		+4.7 (p=0.15)	+8.4 (p=0.002)

The 14B is 8.4 points better on the training split (95% CI +3.2 to +13.6); a gap that grows with model size is what memorisation looks like. The held-out 76.8% is the number comparable to published MBPP results, and the size advantage is +16.2 points on the test split against +19.9 on train. Project 01's regression rate (3.6% test, 3.2% train) does not move. Each project states which split it used; load("mbpp", split="test") selects the held-out 500.

First attempt is very nearly all you get. Projects 02 and 04 reach this from opposite directions: self-debug rounds 1–2 capture 99.3% of what five rounds achieve, and 77.0% of first-attempt failures survive both repair and rewrite.

MBPP's task descriptions are the weak link. Projects 03 and 07 reach this through test suites and through docstrings: a description written back from the code beats MBPP's own by 31.7 points.

Tech stack

• Python 3.11+, standard library for everything except reading parquet
• pandas + PyArrow (the HumanEval parquet; MBPP is read as JSONL)
• Ollama over HTTP (urllib): qwen2.5-coder:3b, qwen2.5-coder:14b, plus qwen2.5:3b-instruct / qwen2.5:14b-instruct for project 16
• subprocess + ast for execution, mutation and refactor checks
• pytest, ruff, uv, GitHub Actions

Run it

git clone https://github.com/hammasbuilds/code-llm-lab
cd code-llm-lab
uv sync --all-groups

uv run pytest -q                                              # 140 tests, no GPU, no Ollama
ollama pull qwen2.5-coder:14b && ollama pull qwen2.5-coder:3b
uv run python projects/01_coder_size_curve/run.py --limit 972  # reproduces project 01

MBPP and HumanEval are read from the local Hugging Face cache (Muennighoff/mbpp, openai/openai_humaneval). Reproducing a measurement needs Ollama with the models pulled; the numbers here were produced on a 16 GB GPU at temperature 0 except project 06.

shared/datasets.py     MBPP and HumanEval normalised into one task shape
shared/execute.py      subprocess with a timeout; pass / fail / error / timeout kept apart
shared/model.py        Ollama over HTTP, with an on-disk generation cache
shared/provenance.py   the config block every results.json carries
projects/NN_*/         run.py · results*.json · README.md
tests/                 shared layer, per-project logic, headline invariants

Licence

MIT, see LICENSE. MBPP (Austin et al., 2021) is CC BY 4.0; HumanEval is MIT.