Benchmarks

Per-model benchmark harness that scores candidate models against Forge's agent prompts. Separate from the Aider polyglot suite (docs/POLYGLOT.md) and the SWE-bench harness (docs/SWE-BENCH.md).

This harness exists to screen models before they go into Forge's live roster — it runs 8 Forge-derived prompts (Director, Red, Green, Fixer, Reviewer, etc.) against each candidate via OpenRouter and scores the raw response with a simple rubric per task.

What gets measured

For each (model, task) pair:

Field	Meaning
ttfb_s	Time-to-first-byte over OpenRouter's streaming endpoint
latency_s	Total wall-clock latency
scores	Per-rubric-item pass/fail dict for that task
score_pct	% of rubric items that passed
raw_response	First 500 characters of the streamed response (for debugging)
error	Exception string if the call failed
input_tokens / output_tokens	Token counts (often null — OpenRouter doesn't always return them)

Results are written to results/ as <provider>_<model>_<timestamp>.json.

Prerequisites

• OPENROUTER_API_KEY set in .env (or the environment). The harness uses OpenRouter exclusively — it does not hit Bedrock.
• Python deps from pip install -e 'platform[dev]' (installs httpx, langchain-openai, python-dotenv).

Commands

Run from the repo root with the venv activated.

# One candidate, all 8 tasks
python -m benchmarks.run_bench moonshotai/kimi-k2.5

# One candidate, one task only (useful while iterating on prompts)
python -m benchmarks.run_bench moonshotai/kimi-k2.5 --task red

# The full candidate sweep (see CANDIDATES in run_bench.py)
python -m benchmarks.run_bench --all

Aggregate and print a summary table:

python -m benchmarks.report

Cost and wall-clock

The full sweep is bounded but not cheap:

• ~8 tasks × ~N candidates = N × 8 LLM calls, each bounded by a 120s timeout.
• Prompts include game-design and repo context, so input-token counts can be large (tens of thousands per call) depending on the task.
• Expect low single-digit USD per model for a full 8-task run on mid-tier models, and longer wall-clocks for larger reasoning models.
• Telemetry in results/*.json captures latency and token counts per call — use these for a precise post-hoc cost estimate.

Treat committed runs as a historical snapshot, not a live leaderboard.

What lives in results/

results/ contains raw JSON outputs from historical sweeps. Each file:

• Covers one model across 8 tasks.
• Is named <provider>_<model>_<timestamp>.json. The _nitro suffix (where present) indicates the OpenRouter :nitro routing variant of the same base model.
• Top-level keys: model, timestamp, results[].

Point-in-time snapshots currently tracked:

• 20260412T* — the main sweep captured on 2026-04-12, covering the candidates listed in CANDIDATES plus ad-hoc additions.

Should I commit new results?

For an external contributor: probably not. The results directory is deliberately kept as a curated historical record. If you're running the harness locally:

• Keep your output files under benchmarks/results/ if you want them tracked, but consider opening an issue describing why the new snapshot is interesting before committing it.
• For one-off analysis, write to a scratch directory outside the repo.

Related

• docs/POLYGLOT.md — Aider polyglot methodology and per-language results.
• docs/SWE-BENCH.md — SWE-bench harness, container mode, and stage-0 runner.
• benchmarks/tasks.py — the 8 task definitions, system prompts, and scorers.
• benchmarks/report.py — summary generator over results/.