inspect_ai evals that ask language models to write Python
validators for two ISO identifiers and generate a message, then score the generated code & messages
against worked examples taken from the corresponding Wikipedia
articles in prose_sources/ or against a known-good message validator. Two batch providers are supported and
can run side-by-side in the same dashboard:
- Doubleword — open-weight frontier models hosted on doubleword.ai, submitted via the OpenAI-compatible Files + Batches API.
- Anthropic — Claude Opus (and any model with a
PRICINGentry inproviders/anthropic.py) submitted via the Message Batches API.
Both share the same task definitions, scorers, schema, and dashboard;
the provider abstraction lives in providers/.
| Eval | What the model produces | Spec source | Scoring |
|---|---|---|---|
iban_eval |
A Python function is_valid_iban(iban: str) -> bool (ISO 13616, mod-97) |
prose_sources/iban_cpy.md |
extract python fence → run against curated test cases in cases.py |
lei_eval |
A Python function is_valid_lei(lei: str) -> bool (ISO 17442, mod-97-10) |
prose_sources/lei.md |
as above, with build_lei_cases |
pain_eval |
A pain.001.001.09 CBPR+ ISO 20022 XML message (SR2025) |
prose_sources/pain_msg_rules_xsd.txt (full XSD + 71-rule CSV, 366 KB / 5 255 lines) |
extract xml fence → pipe through ./validator/cbpr-validate --year 2025 --json - → 1 / (1 + errors + 0.25·warnings + 0.05·infos) |
For code-test evals (iban_eval / lei_eval) the correctness scorer
(scoring.py) extracts the code, executes it in-process, and runs each
generated function against the curated test cases in cases.py. Four
additional static scorers (code_metrics.py) report shape metrics for
the same code, so model comparisons go beyond pass/fail.
For pain_eval the model emits XML instead of Python. pain_scoring.py
extracts the first xml fenced block (falling back to a naked
<?xml … ?> … </Document> slice), pipes it into the
validator/cbpr-validate CLI, and reads the
issue counts (error_count, warning_count, info_count) back as a
single continuous score in [0, 1]:
score = 1 / (1 + errors + 0.25·warnings + 0.05·infos)
A clean message scores 1.0; degraded messages degrade smoothly.
Failure modes that prevent validation at all (no XML extracted, CLI
timeout, transport failure) collapse to 0.0 with a descriptive
failure_class so the dashboard's outcome-breakdown chart can group
them. The validator is the hosted service at
https://cbprstar.com/api/validate; harvest requires network
reachability to that endpoint.
Cost note: the pain spec is ~92k input tokens. With 15 epochs that's
~1.4 M input tokens per (provider, model) eval-run for pain_eval
alone — at Claude Opus 4.7 batch rates ($2.50/MTok in, $12.50/MTok
out) that's ~$3.40 input + variable output. The dashboard's
cost-vs-quality chart reflects this without code changes.
The code-test evals (iban_eval / lei_eval) report five values per
sample; the validator-CLI eval (pain_eval) reports one. Means and
stderrs roll up automatically across samples / models.
| Scorer | Used by | Source | Direction | Meaning |
|---|---|---|---|---|
code_test_scorer |
iban, lei | scoring.py |
higher better (0..1) | fraction of curated test cases the generated function gets right |
pain_validator_scorer |
pain | pain_scoring.py |
higher better (0..1) | 1 / (1 + errors + 0.25·warnings + 0.05·infos) from cbpr-validate |
loc |
iban, lei | radon raw | lower leaner | source lines of code (no blanks/comments) |
complexity |
iban, lei | radon CC | lower simpler | worst-block McCabe cyclomatic complexity |
pep8 |
iban, lei | pycodestyle | lower cleaner | PEP 8 violation count |
maintainability |
iban, lei | radon MI | higher better (0..100) | maintainability index |
If the model returns something that doesn't parse, the static scorers
return 0 with an explanatory note rather than crashing the eval.
pain_eval similarly returns 0.0 with a failure_class of no_xml
when nothing XML-shaped can be extracted from the completion.
| Step | Command | Notes |
|---|---|---|
| 1. Set up | pip install -r requirements.txt then edit .env |
DOUBLEWORD_API_KEY (+ DOUBLEWORD_BASE_URL) for the Doubleword path; ANTHROPIC_API_KEY for the Anthropic path. You only need the ones for providers you intend to run. |
| 2a. Doubleword batch run | ./run_evals.sh |
1h batch (default). MODE=24h ./run_evals.sh for cheaper, slower. ./run_evals.sh realtime for synchronous. |
| 2b. Anthropic batch run | ./run_evals.sh anthropic |
24h batch on Claude Opus (the only window Anthropic offers). Override the model list with ANTHROPIC_MODELS="claude-opus-4-7". |
| 3. Wait + harvest | ./collect.sh --watch |
Polls every 60s until pending batches resolve (both providers in one pass), writes .eval logs into logs/, and populates dwevals.db as part of the same step. |
| 4. (Optional) rebuild inspect bundle | ./plot.sh |
Rebuilds the inspect_ai SPA bundle so the "open →" links on the Runs page resolve. Re-run after harvesting fresh logs if you want to click into them; skippable otherwise. |
| 5. View the dashboard | python server.py |
Charts, filters, runs table at http://127.0.0.1:8000/. Charts hit per-card SQL endpoints under /api/chart/<name> — filter toggles trigger fresh queries. |
| 6. Ad-hoc SQL | sqlite3 dwevals.db "…" or open in DBeaver |
Schema docs in db_schema.sql. See Querying the database below. |
| 7. HTTP API reference | open http://127.0.0.1:8000/api/docs | Live reference for every /api/* JSON endpoint. Checked-in copy in docs/api.md. Regenerate with python -m api_docs --markdown > docs/api.md. |
| Goal | Command |
|---|---|
Rebuild dwevals.db from scratch |
python collect.py --backfill-db (idempotent — re-running over unchanged files is a no-op) |
Re-stamp window (1h/24h) onto historic .eval logs |
python collect.py --backfill-window |
Re-fetch Doubleword /batches/{id}/analytics and re-stamp cost_usd onto historic .eval logs |
python collect.py --backfill-cost |
| One-off realtime run for prompt iteration | inspect eval iban_eval.py --model openai-api/doubleword/<model-id> |
./collect.sh writes the DB automatically during the harvest path —
the only time you need the backfill commands is when you've
hand-edited logs, dropped dwevals.db, or added new fields the older
logs don't carry.
sudo apt install pandoc # only needed if you re-fetch the Wikipedia sources
pip install -r requirements.txt
cp .env.example .env
# edit .env and set:
# DOUBLEWORD_API_KEY (+ DOUBLEWORD_BASE_URL if non-default) for the Doubleword provider
# ANTHROPIC_API_KEY for the Anthropic provider
# You only need credentials for the providers you actually want to run.Doubleword exposes an OpenAI-compatible endpoint, so we use
inspect_ai's generic openai-api/<provider>/<model-id> adapter.
inspect_ai derives the env-var names from the provider segment, so it
reads DOUBLEWORD_API_KEY and DOUBLEWORD_BASE_URL automatically.
The Anthropic provider talks to https://api.anthropic.com via
x-api-key auth; the Python SDK isn't a dependency (we use httpx
directly to keep the provider symmetric with batch_client.py).
The model id is tenant-specific. To list the models available on your endpoint:
curl "$DOUBLEWORD_BASE_URL/models" -H "Authorization: Bearer $DOUBLEWORD_API_KEY"inspect eval iban_eval.py --model openai-api/doubleword/<model-id>
inspect eval lei_eval.py --model openai-api/doubleword/<model-id>Browse the resulting logs (prompt, completion, extracted code, per-case pass/fail):
inspect viewUseful flags: --epochs N re-runs each sample N times for variance,
--temperature 0.2 controls sampling, --limit 1 runs a single sample.
Run both evals against each model in turn — every run drops a fresh
.eval log into logs/, ./collect.sh writes the same data into
dwevals.db, and the dashboard reads from the DB.
./run_evals.sh # default: 1h batch — submits and exits
./collect.sh --watch # poll until batches complete; harvest into logs/ + DB
./plot.sh # (optional) rebuild inspect_ai bundle for the "open →" links
python server.py # browse at http://127.0.0.1:8000/The default model list lives at the top of run_evals.sh. Override
either step:
MODELS="moonshotai/Kimi-K2.6" ./run_evals.sh # re-run one model only
LOG_DIR=other-logs BUNDLE_DIR=other-bundle ./plot.sh # bundle from a different log dirDoubleword bills batched jobs at a discount. run_evals.sh supports
three modes — pass as the first argument or the MODE env var:
| Mode | Cost | Latency | Behaviour |
|---|---|---|---|
1h (default) |
cheaper | up to 1 hour | submit batch and exit; harvest with ./collect.sh |
24h |
cheapest | up to 24 hours | same as 1h, longer window |
realtime |
full price | seconds | block on inspect eval like before — best for prompt iteration |
./run_evals.sh # 1h batch
./run_evals.sh realtime # synchronous (or MODE=realtime ./run_evals.sh)
MODE=24h ./run_evals.sh # cheapest
./collect.sh --watch # poll until pending batches resolvePending batch manifests live under batches/; completed ones move
to batches/done/. collect.sh writes native .eval logs into
logs/ and upserts them into dwevals.db in the same step, so
the home page picks them up on the next refresh.
plot.sh now only rebuilds the inspect_ai bundle under
inspect_bundle/ so the "open →" links on the runs page resolve.
Run it after ./collect.sh only if you want to click into recently
harvested logs in the inspect_ai viewer. The legacy per-(task,
scorer) PNGs are no longer built by default — run
python plot_results.py logs -o plots directly if you want fresh
ones for embedding.
What dwevals sets explicitly, and what it leaves at the API/model default. The goal is uniformity — every model gets the same input shape, so cross-model differences reflect the model, not the request.
The three knobs every model receives, defined as constants in
submit_batch.py:27-32 and imported by iban_eval.py /
lei_eval.py so realtime and batch paths agree:
| Parameter | Value | Defined at | Applied to |
|---|---|---|---|
reasoning_effort |
"high" |
submit_batch.py:30 (REASONING_EFFORT) |
Batch request body (submit_batch.py:116); realtime GenerateConfig in iban_eval.py:91 and lei_eval.py:91; stamped onto the batch manifest at submit_batch.py:152 so the runs table can show what was requested. |
max_tokens |
120000 |
submit_batch.py:31 (MAX_TOKENS) |
Batch request body; realtime GenerateConfig. Acts as a fail-fast cap rather than a target — reasoning models can spend many thousands of tokens thinking before emitting the function. |
timeout (seconds) |
300 |
submit_batch.py:32 (GEN_TIMEOUT_S) |
Realtime GenerateConfig only. Batch jobs are governed by the 1h/24h window instead. |
Plus two run-shape constants:
| Parameter | Value | Defined at | Notes |
|---|---|---|---|
epochs |
15 |
iban_eval.py:81, lei_eval.py:82, submit_batch.py:61 (CLI default) |
Each prompt is sent 15× so every run carries a real distribution. Override per-submit with --epochs N. |
| Eval-script concurrency | --max-connections 1, --fail-on-error 1.0 |
run_evals.sh:48-49 |
Realtime path only. Serialises requests per model and aborts the eval if every sample errors. |
reasoning_effort = "high" is the strongest setting all five
Doubleword reasoning models we test share semantically (Qwen / Kimi /
GLM map high to "thinking enabled"; DeepSeek treats it as native;
Mistral's Devstral-2 has no reasoning knob and surfaces a 0/N honor
badge by design). See the methodology bullet on the home page for the
full explanation.
Both providers receive the same abstract reasoning_effort = "high"
on the manifest (so the runs table reads consistently), but each
provider translates it into the right shape for its API:
| Provider | What goes on the wire |
|---|---|
doubleword |
reasoning_effort: "high" in the request body, sent verbatim — providers/doubleword.py:DoublewordProvider.submit |
anthropic |
thinking: {type: "adaptive"} in the per-request params block — providers/anthropic.py:_thinking_for. Opus 4.7 only accepts "adaptive" and lets Anthropic pick the thinking budget; older Opus / Sonnet / Haiku models accept "adaptive" too, so we use it uniformly. The string "high" is recorded for cross-provider comparison but does not influence the budget — that's chosen by Anthropic. |
This is the same asymmetry the Doubleword side already carries
(Devstral-2 has no reasoning knob and shows 0/N); the Opus 4.7
column will show whatever honor count Anthropic returns based on
how often it engaged thinking blocks on a given sample.
runs.cost_usd for Anthropic rows is computed client-side from
per-row usage.{input_tokens, output_tokens} × the rate card baked
into providers/anthropic.py:PRICING:
| Model | Batch input ($/MTok) | Batch output ($/MTok) |
|---|---|---|
claude-opus-4-7 |
2.50 | 12.50 |
(50% batch discount already applied — these are the rates the
endpoint actually bills at.) Adding Sonnet or Haiku is a one-line
edit to PRICING; thinking tokens are rolled into output_tokens
by Anthropic so no separate cost line is needed. Doubleword rows
keep using the authoritative /batches/{id}/analytics figure rather
than a pricing card.
These parameters are not set anywhere in the request body or
GenerateConfig — each model uses its own default. We've kept it
that way because what counts as a "sensible" default differs per
model, and pinning a value would silently advantage the models whose
default already matches it.
| Parameter | Why we don't set it |
|---|---|
temperature |
Each model has its own preferred default; pinning would bias toward those that already match. We rely on epochs=15 to expose sampling noise. |
top_p |
Same reason as temperature. |
top_k |
Not all providers expose it; not portable across the model set. |
frequency_penalty / presence_penalty |
No reason to bias against / for repetition for code-gen. |
seed |
Determinism would mask the real variance the boxplots show. |
stop (stop sequences) |
Not needed — the prompt asks for a single fenced code block and the scorer extracts the first one. |
response_format (JSON schema / structured output) |
Models behave very differently when forced into JSON mode; we keep the prompt natural. |
logit_bias, n, stream |
All defaults. We always get one completion per request, non-streamed. |
If you want to add a new pinned parameter, treat it like
reasoning_effort: add a constant to submit_batch.py:27-32,
reference it from the batch request body and both eval
GenerateConfigs, and stamp it onto the manifest so the runs table
can show what was requested.
python server.py # http://127.0.0.1:8000/The home page is a single-page local viewer driven by SQL queries
against dwevals.db. Filters at the top — Time window,
Execution mode (realtime / 1h batch / 24h batch), Models
— re-issue the chart queries each time you toggle, so every card
shows the same slice consistently. The chart types are:
- Snapshot bars — latest run per model for each (task, scorer).
- Time-series line charts — mean ± 1 σ across each run's epochs, with a per-model tab strip on each card.
- Cost vs quality — output tokens vs test-case pass rate per task.
- Actual cost vs quality — combined USD cost (from
Doubleword's
/batches/{id}/analytics) vs mean pass rate per model. - Outcome breakdown — stacked bar of passed / partial / wrong / exec error / timeout / no code across the latest run's epochs.
- Per-test-case heatmap — pass rate per curated input × model.
The runs table lives on a separate page at http://127.0.0.1:8000/runs
(linked from the bottom of the home page). Each row links into
inspect_ai's run viewer at /inspect/?log_file=…, served from the
static bundle that ./plot.sh regenerates. The viewer is a SPA that
fetches the .eval at view time — first paint takes a moment. When
the bundle is stale (a log is missing because ./plot.sh hasn't been
re-run since logs changed) the row renders a "stale (run ./plot.sh)"
hint instead of a broken link.
Failed runs in the table carry a small no code / exec err /
no fn badge, distinguishing extraction failures from
genuinely-wrong code, and a reasoning honor badge (e.g. high · 0/15)
when fewer than every epoch returned reasoning tokens.
The eval modules read fixed paths:
prose_sources/iban_cpy.md(Wikipedia IBAN article, trimmed)prose_sources/lei.md(Wikipedia LEI article, trimmed)prose_sources/pain_msg_rules_xsd.txt(CBPR+ pain.001.001.09 XSD + 71-rule business CSV — checked in verbatim from the SR2025 release; ~366 KB / ~92k input tokens)
To regenerate the IBAN/LEI markdown from Wikipedia, run
./get_wikis.sh (which writes timestamped files) and then trim/copy
the relevant content into iban_cpy.md / lei.md. Trimming matters:
the full Wikipedia IBAN article is ~78k tokens and will hang or
overflow many models.
The pain spec is not auto-regenerated — replace it manually when a new
CBPR+ SR is published. The validator CLI's --year flag determines
which rule-year the message is checked against (pain_eval.py pins
this to 2025).
iban_eval.py # @task iban_eval — IBAN spec + dataset + scorer wiring
lei_eval.py # @task lei_eval — LEI spec + dataset + scorer wiring
pain_eval.py # @task pain_eval — CBPR+ pain.001 XML + validator scorer
scoring.py # correctness scorer: extracts code, execs it, runs test cases
pain_scoring.py # validator scorer for pain_eval: extracts XML, runs cbpr-validate
code_metrics.py # static scorers: loc, complexity, pep8, maintainability
validator/cbpr-validate # CLI client for cbprstar.com — XSD + CBPR+ business rules
plot_results.py # reads logs/ and writes plots/*.png (legacy)
run_evals.sh # loops models and runs both evals on a chosen provider
submit_batch.py # provider-agnostic batch submitter (--provider doubleword|anthropic)
collect.py # harvests completed batches → .eval logs + dwevals.db
plot.sh # rebuilds inspect_ai bundle for the "open →" links
server.py # local web viewer; serves /api/chart/* SQL endpoints
api.py # legacy full-payload /api/runs.json
api_charts.py # per-chart SQL endpoints (snapshot, history, …)
api_docs.py # source-of-truth for docs/api.md and /api/docs
docs/api.md # generated HTTP-API reference (run python -m api_docs)
docs/schema.svg # ER diagram of dwevals.db (hand-drawn from db_schema.sql)
db_ingest.py # populates dwevals.db from logs/ + batches/done/
db_read.py # the SQL-backed read path the server uses
db_schema.sql # SQLite DDL for the derived index
batch_client.py # OpenAI-compatible Files/Batches client (used by providers/doubleword.py)
providers/__init__.py # get_provider(name) factory
providers/base.py # Provider Protocol + SubmitRequest/BatchStatus/SampleResult dataclasses
providers/doubleword.py # adapter wrapping batch_client.py
providers/anthropic.py # adapter for Anthropic /v1/messages/batches + PRICING card
cases.py # curated valid examples + mutation helper for negatives
webui/ # frontend (app.js, plotly.min.js)
prose_sources/ # Wikipedia markdown (regenerable via get_wikis.sh)
.env.example # DOUBLEWORD_API_KEY / ANTHROPIC_API_KEY placeholders
Run data lives in dwevals.db — a SQLite index built from the .eval
log files on every harvest. The logs remain the authoritative source;
the DB is rebuildable at any time:
python collect.py --backfill-db
Open the file directly in DBeaver, the sqlite3 CLI, or any other
SQLite client. The schema is documented in db_schema.sql; a visual
ER diagram lives at docs/schema.svg. Key tables:
runs (one row per .eval log, indexed by model / provider / mode /
created_at), run_scores (one row per (run, scorer) — long-form so
arbitrary scorers fit without schema changes), samples (per-epoch
rows with failure_class, token counts, and a per_case_json blob),
batches (submission metadata + total cost / token rollups — read
from Doubleword's /batches/{id}/analytics endpoint on Doubleword
batches; computed client-side from Anthropic's per-row usage block
on Anthropic batches).
The provider column on both runs and batches distinguishes
doubleword from anthropic rows; the schema migrated from v1 to
v2 with the Anthropic integration and all pre-existing rows were
backfilled to provider = 'doubleword'.
runs.mode is one of realtime, batch_1h, or batch_24h, so you
can compare execution modes directly:
-- Cost per model per mode
SELECT model, mode, ROUND(AVG(cost_usd), 4) AS mean_usd, COUNT(*) AS n
FROM runs
WHERE cost_usd IS NOT NULL
GROUP BY model, mode
ORDER BY model, mode;
-- Latest run per model
SELECT r.model, r.task, r.mode, r.created_at, r.cost_usd
FROM runs r
JOIN (
SELECT model, MAX(created_at) AS m FROM runs GROUP BY model
) latest ON r.model = latest.model AND r.created_at = latest.m
ORDER BY r.model;
-- Token throughput per (model, mode)
SELECT model, mode,
SUM(input_tokens) AS in_toks,
SUM(output_tokens) AS out_toks,
COUNT(*) AS n
FROM runs
WHERE input_tokens IS NOT NULL
GROUP BY model, mode
ORDER BY model, mode;