Claude Code Benchmark

A research benchmark for comparing coding agents on identical tasks.

This repository runs the same Claude Code CLI against two backends — native Anthropic and Free Claude Code + Z.ai — then scores the resulting workspaces with an objective eight-dimension pipeline (build, tests, E2E, rubric, and more). It is designed for reproducible experiments: frozen artifacts, re-scoring without re-running models, and cross-task rollups.

→ Full documentation
→ Benchmark results (July 2026) — published run: 5/6 tasks both PASS, 1 model gap on greenfield

Documentation

Concepts

Topic	Link
What is a harness?	docs/concepts/harness.md
Arms & controlled comparison	docs/concepts/arms.md
Tasks & prompts	docs/concepts/tasks.md
Reports & artifacts	docs/concepts/reports.md

Analyses (scoring dimensions)

Dimension	Weight	Link
Overview	—	docs/analyses/overview.md
Build & unit tests	20%	docs/analyses/build-and-unit-tests.md
E2E & accessibility	25%	docs/analyses/e2e-and-accessibility.md
Code quality	15%	docs/analyses/code-quality.md
Architecture rubric	15%	docs/analyses/architecture-rubric.md
Acceptance criteria	10%	docs/analyses/acceptance-criteria.md
UI quality	5%	docs/analyses/ui-quality.md
Workspace hygiene	5%	docs/analyses/workspace-hygiene.md
Efficiency	5%	docs/analyses/efficiency.md
Model quality rollup	—	docs/analyses/model-quality-rollup.md

Research

Topic	Link
Methodology & validity	docs/research/methodology.md
Why our tests are reliable	docs/research/reliability.md
Model comparison (Sonnet vs GLM)	docs/research/model-comparison.md
Published benchmark results	reports/BENCHMARK_RESULTS.md

---

Quick start

git clone https://github.com/Shhb-Coder-1999/claude-code-benchmark.git
cd claude-code-benchmark

py -3 -m venv .venv
.venv\Scripts\Activate.ps1
py -3 -m pip install -e .

cd bench/e2e
npm install
npx playwright install chromium
cd ../..

Secrets

Only the FCC+Z.ai arm needs a Z.ai API key:

copy .env.example .env.local
# Edit .env.local — set ZAI_API_KEY only

Paste the same ZAI_API_KEY into FCC Admin UI → Providers → Validate → Apply.

Preflight check

py -3 -m harness.compare --preflight

Usage

Single task, both arms

py -3 -m harness.compare --task 00-harness-smoke

Full suite (3 runs per task)

py -3 -m harness.compare --suite all --runs 3

One arm only

py -3 -m harness.run --task 01-greenfield-todo --arm native
py -3 -m harness.run --task 01-greenfield-todo --arm fcc_zai

Z.ai rate limits (HTTP 429)

Free-tier models can return rate-limit errors. The harness applies cooldowns, retries, and marks blocked runs as INFRA (not a code-quality fail). See arms doc and bench/config/benchmark.yaml.

Tasks

ID	Description
00-harness-smoke	Tiny TS + Vitest sanity check
01-greenfield-todo	Vue 3 todo app from spec
02-bugfix	Fix 3 bugs in a Vue 3 todo app
03-feature-add	Add due dates + sorting (medium)
04-small-feat	Clear completed + active count (small)
05-heavy-feat	Categories + search + export/import (heavy)

Task layout: bench/tasks/<id>/spec/ + starter/. Details in docs/concepts/tasks.md.

Reports

Results land in reports/<run_id>/ — see docs/concepts/reports.md.

Published snapshot: reports/BENCHMARK_RESULTS.md (committed; per-run folders remain gitignored).

py -3 -m harness.serve          # hub API for re-score / re-run
py -3 -m harness.rescore --run-id <run_id> --arm both
py -3 -m harness.scripts.model_quality_report

Project structure

claude-code-benchmark/
├── docs/                       # Research documentation (start here)
├── bench/                      # Tasks, config, E2E, tooling
├── harness/                    # Python benchmark engine
├── dashboard/                  # Optional Vue report viewer
└── reports/                    # Runtime artifacts (gitignored); see BENCHMARK_RESULTS.md

Prerequisites

1. Claude Code CLI — claude --version works (native arm uses CLI login, no ANTHROPIC_API_KEY in repo).
2. Free Claude Code — install guide.
3. FCC + Z.ai — fcc-server, Admin UI, ZAI_API_KEY.
4. Python 3.11+ and Node.js.

Helper scripts

Command	Purpose
py -3 -m harness.scripts.validate_scorer	Smoke-test the scorer
py -3 -m harness.scripts.rescore_all	Batch re-score all runs
py -3 -m harness.scripts.list_suspended	List non-OK runs
py -3 -m harness.scripts.model_quality_report	Cross-task comparison HTML

License

MIT — see LICENSE.