cuad-serving-bench

How fast, how accurate and how cheap is it to self-host a fine-tuned contract-review model? A Qwen3-4B model fine-tuned on CUAD (41 legal clause types, model on Hugging Face) benchmarked under 5 serving setups, load-tested up to 32 simultaneous users, priced per 1,000 contracts, compared with Claude Opus 5.5, and packaged as a one-command private service.

Live demo (free GPU, no install): huggingface.co/spaces/Ihteshamstar/cuad-clause-extractor - paste a contract excerpt and get the 41 clause types back in ~10-30 seconds.

Results at a glance

• The fine-tuned 4B model beats Claude Opus 5.5 zero-shot: F1 0.900 vs 0.869 (paired bootstrap, difference -0.031, 95% CI -0.048 to -0.014), at ~1/50 to 1/130 of the cost and 3x faster per contract.
• FP8 is the best way to serve it: same accuracy as 16-bit (no measurable difference), 45% less memory, and the highest throughput: ~36 contract chunks per minute (~2,100 per hour) on one RTX 5090.
• vLLM vs Ollama: equal for 1 user, but Ollama hits a wall at its 4-6 parallel slots. At 16 users vLLM FP8 does 2.2x the throughput with the first token in 0.8 s instead of 41 s.
• Two 4-bit formats, two outcomes: Ollama's Q4_K_M kept full accuracy; vLLM's AWQ lost 1.5 F1 points (statistically real) and looped most often under load.
• Repetition loops are the hidden cost: 5-9% of requests loop to the 12,288-token cap; on the vLLM runs they used 44-57% of all generated tokens, roughly doubling the cost per contract.

Questions this answers

1. How much accuracy is lost when the model is compressed (FP8, 4-bit)? (Phase 1)
2. How many contracts per minute can one GPU handle? (Phase 3)
3. What happens when 1 / 4 / 16 / 20 / 32 people use it at once? (Phase 3)
4. How much do 1,000 contracts cost, compared with frontier-model APIs? (Phase 4)

Setups compared

ID	Engine	Weights	Size
A	Ollama	F16 GGUF	~8 GB
B	Ollama	Q4_K_M GGUF	~2.5 GB
C	vLLM	BF16	7.5 GiB
D	vLLM	FP8 (llm-compressor, dynamic)	4.1 GiB
E	vLLM	AWQ 4-bit (W4A16, 128 train-chunk calibration)	2.5 GiB

Same model, same 150 test chunks, same prompts, temperature 0, same fallback rules for every setup: docs/PROTOCOL.md.

Architecture of the deployable service

flowchart LR
    U["Client<br/>(contract text)"] -->|"POST /extract<br/>optional X-API-Key"| A["FastAPI service<br/>port 8080"]
    A -->|"split: 60k-char chunks,<br/>8k overlap"| C{{"parallel chunk requests<br/>(max 8)"}}
    C -->|"OpenAI-compatible API,<br/>JSON mode, temp 0"| V["vLLM 0.30<br/>FP8 Qwen3-4B<br/>(GPU, internal only)"]
    V -->|"loop or invalid JSON"| R["retry ladder<br/>penalty 1.05 / 1.15 / 1.25"]
    R --> V
    V --> M["merge quotes per clause,<br/>report failed chunks"]
    M -->|"41 clause types + timing"| U

Loading

Run it: private contract clause extraction

serve/ packages the best setup from the benchmark (vLLM + FP8) behind a small API:

# put the FP8 model folder at serve/models/cuad-4b-s42-fp8 (made with bench/quantize.py fp8)
cd serve
docker compose up -d --build                 # vLLM on the GPU + FastAPI on port 8080
python try_extract.py path/to/contract.txt   # or: POST /extract {"text": "..."}

• POST /extract: full contract text in, all 41 CUAD clause types out (verbatim quotes), with the number of chunks, any chunk that failed, and the time taken. Long contracts are split exactly like training (60,000-character chunks, 8,000 overlap) and processed in parallel; loops are retried with the benchmark's fallback ladder (repetition penalty 1.05, 1.15, 1.25).
• GET /health, GET /clauses. Optional API_KEY env var requires an X-API-Key header.
• Only the API port is published; the model server stays inside the Compose network.
• CI (GitHub Actions): lint, 12 API tests with a fake model, Docker build and a container smoke test. GitHub's runners have no GPU, so the GPU benchmarks are run manually.

Layout

bench/     benchmark scripts: accuracy runs, quantization, load tester, cost model, Claude baseline, charts
serve/     the deployable service: FastAPI app, Dockerfile, docker-compose.yml, try_extract.py
tests/     API tests (fake model; run in CI)
results/   accuracy/, load/, claude/, raw/ - measurement files, never overwritten
docs/      protocol, charts, plain-language explainer

Results

Accuracy (150 test chunks, 102 contracts, full fallback ladder 1.05 → 1.15 → 1.25)

Setup	Detection F1 (95% CI)	Precision	Recall	Token overlap on TP	Verbatim rate	Wall time
A: Ollama F16	0.897 (0.884–0.910)	0.916	0.879	0.947	0.972	46.7 min, sequential
B: Ollama Q4_K_M	0.900 (0.887–0.913)	0.901	0.900	0.938	0.972	~35–40 min, sequential
C: vLLM BF16	0.900 (0.888–0.913)	0.921	0.880	0.948	0.955	17.8 min, concurrency 16
D: vLLM FP8	0.895 (0.883–0.908)	0.923	0.869	0.951	0.970	13.6 min, concurrency 16
E: vLLM AWQ 4-bit	0.885 (0.870–0.900)	0.904	0.866	0.933	0.971	18.4 min, concurrency 16

Ollama F16, FP8 and AWQ each left 1 of 150 chunks unrecovered after the full ladder (scored as "found nothing", --allow-partial). F16 and FP8 lost the same Monsanto chunk; AWQ lost a Goosehead chunk.

Paired comparison vs vLLM BF16 (bench/compare_accuracy.py, bootstrap over contracts):

Setup	F1 diff, all 150 chunks (95% CI)	F1 diff, 148 common chunks (95% CI)
A: Ollama F16	−0.0031 (−0.0116 to +0.0025)	+0.0004 (−0.0029 to +0.0042)
B: Ollama Q4_K_M	+0.0001 (−0.0076 to +0.0072)	−0.0003 (−0.0077 to +0.0069)
D: vLLM FP8	−0.0049 (−0.0131 to +0.0022)	−0.0014 (−0.0061 to +0.0033)
E: vLLM AWQ 4-bit	−0.0155 (−0.0247 to −0.0067)	−0.0155 (−0.0242 to −0.0068)

Ollama F16, Ollama Q4_K_M and FP8: intervals contain 0, no measurable difference. AWQ: interval excludes 0, a real loss.

Correction: the Ollama row was first labelled "F16". The Ollama model cuad-extractor was built with ollama create --quantize q4_K_M (confirmed with ollama show), so it is setup B.

	BF16	FP8	AWQ 4-bit
Weights	7.49 GiB	4.11 GiB (−45%)	2.48 GiB (−67%)
KV cache at --gpu-memory-utilization 0.85	125,360 tokens	150,048 (+20%)	161,520 (+29%)
vLLM linear kernel	default	CUTLASS FP8	Marlin W4A16
Primary pass (150 chunks)	11.0 min	8.6 min	9.9 min
Median latency, successful requests	27.6 s	24.3 s	24.1 s
Primary-pass loops (hit 12,288 tokens)	10	9	14
Output tokens wasted in loops	122,880 (47%)	110,592 (44%)	172,032 (57%)

Raw results: results/accuracy/ (eval_cuad-f16_short.json = setup A, eval_cuad-extractor_short.json = setup B).

Early findings:

• Same accuracy, ~2× faster end to end. vLLM matches Ollama's F1 exactly. 17.8 min total (primary pass 11.0 min + fallback retries 6.8 min) vs ~35–40 min for the Ollama queue job. The Ollama figure comes from the old queue logs and includes model loading, so it is approximate. The load tests in Phase 3 give the exact comparison.
• Repetition loops cost almost half the compute. On vLLM's primary pass, 10 of 150 requests (7%) looped to the 12,288-token cap: 122,880 wasted output tokens vs 140,158 useful ones from the other 140. Median time: 276 s for a looping request vs 28 s for a normal one.
• vLLM needed the fallback ladder more: 10 primary failures → 5 after 1.05 → 2 after 1.15 → 0 after 1.25. Ollama: 9 → 2 → 0.
• vLLM's repetition penalty hurts verbatim quoting. It penalizes every token already seen, including the contract text in the prompt; Ollama's only covers the last 64 generated tokens. Verbatim rate on fallback-decoded chunks: 0.841 on vLLM vs 0.973 on Ollama. On normally decoded chunks vLLM is 0.977 vs 0.973. The whole overall verbatim gap comes from the ~10 fallback chunks.
• The engine does not change accuracy. Ollama F16 vs vLLM BF16 on the chunks both answered: +0.0004 F1, 95% CI −0.003 to +0.004. Same model at 16-bit, two engines, same answers.
• FP8 costs no measurable accuracy. Its −0.005 F1 gap is mostly one Monsanto chunk that still looped at penalty 1.25 (16 clause types in the gold answer). Ollama F16 lost the same chunk at full precision, so that is a hard chunk, not an FP8 weakness. On common chunks FP8 is −0.0014. F1 recovery 99.5% (99.8% on common chunks), in line with published FP8-dynamic results (99.3% at 8B in Kurtic et al. 2024; ~100% for Red Hat's Qwen3-14B-FP8-dynamic).
• AWQ 4-bit costs real accuracy: −1.5 F1 points, statistically clear (95% CI −2.5 to −0.7). Precision drops more than recall (0.904 vs 0.921; 0.866 vs 0.880), quote overlap drops (0.933 vs 0.948), and it writes more malformed clause names (25 dropped keys vs 4).
• Smaller is not faster here. AWQ's per-request speed equals FP8's (median 24.1 s vs 24.3 s), but it loops more (14 vs 9 on the primary pass), so 57% of its generated tokens were wasted and the full run took longer than FP8 (18.4 vs 13.6 min). Its only win is memory: +8% KV cache over FP8.
• So far FP8 is the best trade-off on vLLM: same accuracy as BF16, 45% less memory, 22% faster.
• Two 4-bit formats, two different outcomes. Ollama's Q4_K_M matches both 16-bit setups (0.900), while vLLM's AWQ 4-bit loses 1.5 points at a similar file size. Since the engine itself makes no difference (above), the gap points to the quantization method: Q4_K_M is a mixed format that keeps some tensors at 6-bit. (Calibration length is an unlikely cause: 2,048 tokens and 128 samples is the llm-compressor recommendation, and AWQ is reported to be insensitive to calibration data.) The size of the AWQ loss is in line with published results: 98.3% F1 recovery here vs 98.7% average recovery for 4-bit weight-only on 8B models in Kurtic et al. 2024 ("Give Me BF16 or Give Me Death"), with smaller models expected to lose slightly more. The same paper finds GPTQ beats AWQ on real-world tasks, so GPTQ is the obvious next 4-bit method to try (future work).
• For a batch job, vLLM is about 2.3× faster than Ollama end to end at the same F16/BF16 precision: 17.8 vs 46.7 min for all 150 chunks including fallbacks, but the Ollama time includes ~5.3 min of client connection delay (localhost on this machine adds ~2 s per connection; see PROTOCOL.md), so about 41.4 min of real work. This mixes engine speed with concurrency (16 parallel requests vs one at a time); the Phase 3 load tests separate the two.

Load tests: many users at once (Phase 3)

64-chunk speed set, each level run twice; full tables in results/load/load_summary.md (built by bench/summarize_load.py). Chunks per minute, mean of 2 runs (loop-free runs in brackets):

Setup	1 user	4	16	20	32
vLLM BF16	7.6	19.2	29.2	29.7	27.2
vLLM FP8	9.9	24.0	35.0	32.8	36.7
vLLM AWQ 4-bit	10.7	23.1	29.0 (35.4)	30.3 (35.6)	35.9
Ollama F16, 4 slots	8.1	15.9	16.1	16.1	16.4
Ollama Q4_K_M, 6 slots	13.8	13.7	20.8 (23.2)	22.5	19.8 (22.5)

Typical wait (p50) at 16 users: FP8 17 s, BF16 22 s, AWQ 18 s, Ollama Q4 39 s, Ollama F16 54 s. Time to first token at 16 users: FP8 0.8 s, BF16 2.5 s, Ollama Q4 24 s, Ollama F16 41 s.

• One user: Ollama Q4_K_M is fastest (13.8 chunks/min, typical wait 3.3 s). Its 2.5 GB of weights are the least to read per generated token (290 tok/s per request vs 147 for F16).
• Many users: vLLM wins, and the gap grows with load. At 16-32 users vLLM FP8 does 35-37 chunks/min (about 2,100 per hour), 2.2x Ollama F16 and about 1.6x Ollama Q4 (loop-free).
• Ollama hits a hard wall at its slot count. It reserves a full 28,672-token KV slot (3.9 GiB) per parallel request, so only 4 (F16) or 6 (Q4) fit on 32 GB. Beyond that requests queue: at 32 users the first token takes 87 s (F16) or 55 s (Q4). vLLM shares one KV pool, so about 15-19 average requests run at once.
• vLLM BF16 saturates at about 16 users, matching the KV-cache estimate (125,360 tokens / ~8,400 per request). FP8 and AWQ have more KV room and peak about 20% higher.
• Ollama Q4 barely benefits from parallel requests: 4 users get the same total throughput as 1 (per-request decode drops from 290 to 58 tok/s). F16 nearly doubles. Likely llama.cpp's quantized kernels batch less efficiently at small batch sizes (not verified).
• 4-bit formats loop under load. The speed set excludes every chunk that looped in Phase 1, yet AWQ looped in 6 of 10 runs and Q4_K_M in 2 of 10 (on chunks that had not looped before), while BF16, FP8 and Ollama F16 had 0 loops in 30 runs. One loop (12,288 tokens, 2-3 minutes) cuts a run's throughput by about a third.
• Recommendation: a single user on a desktop: Ollama Q4_K_M. A team or a service: vLLM FP8.

Cost and a frontier-model baseline (Phase 4)

Cost per 1,000 contracts (1 contract = 1.47 chunks, as in the CUAD test set). Self-hosted = RTX 5090 rental price / measured throughput at full load (bench/cost.py, results/cost.json). "Realistic" adds back the loop and retry work measured in Phase 1; "best case" assumes loops are fixed.

Option	F1	USD per 1,000 contracts	Typical wait, 1 user
vLLM FP8, self-hosted	0.895	0.57-1.42 realistic (0.27-0.66 best case)	4.5 s
vLLM BF16, self-hosted	0.900	0.82-2.03 realistic	5.9 s
Ollama Q4_K_M, self-hosted	0.900	0.83-2.06 realistic	3.1 s
GPT-4.1 mini API (price only, accuracy not measured)	-	7.40	-
GPT-4.1 API (price only, accuracy not measured)	-	36.98	-
Claude Opus 5.5 API, zero-shot (measured)	0.869	75.54 batch / 151.07 normal	13.2 s

GPU prices: $0.40 (cheapest Vast.ai), $0.73 (Vast.ai median), $0.99 (RunPod on-demand) per hour, Oct 2026. API prices from the providers' pricing pages, Oct 2026. Wait times: the 5 speed-set chunks, one at a time.

Claude Opus 5.5 baseline (bench/claude_baseline.py): all 150 test chunks, same zero-shot protocol as the earlier Qwen3-14B baseline (system prompt + the 41 official clause definitions + chunk, JSON requested in the prompt), Batch API, effort medium, hard $13 budget guard. Total cost $7.91 (147 batch requests incl. 2 retries + 5 latency requests at normal price). A strict 41-key JSON schema was rejected by the API ("compiled grammar is too large"), so JSON is requested in the prompt; 2 answers contained raw line breaks inside JSON strings and succeeded on the automatic retry.

	Fine-tuned Qwen3-4B (vLLM BF16)	Claude Opus 5.5 zero-shot	Qwen3-14B zero-shot
Detection F1 (95% CI)	0.900 (0.888-0.913)	0.869 (0.854-0.883)	0.816
Precision / Recall	0.921 / 0.880	0.803 / 0.948	0.814 / 0.818
Quote overlap with lawyers' spans	0.948	0.872	-
Verbatim rate	0.955	1.000	-
F1 difference vs fine-tuned (paired, 95% CI)	-	-0.031 (-0.048 to -0.014)	-0.084 (-0.098 to -0.070)

• The fine-tuned 4B model beats Claude Opus 5.5 by 3.1 F1 points (statistically clear), at about 1/50 to 1/130 of the cost (batch API vs FP8 realistic) and 3x faster for a single user.
• They make different mistakes. Claude finds more clauses (recall 0.948 vs 0.880) but also flags many the lawyers did not (precision 0.803 vs 0.921), and its quote boundaries match the lawyers' less (0.872 vs 0.948), although every quote it gives is a perfect copy (verbatim 1.000). Fine-tuning taught the 4B model CUAD's annotation conventions (what counts as a clause, where a quote starts and ends); a zero-shot model only has the one-line definitions.
• Caveats: zero-shot only (few-shot examples or prompt tuning could narrow the gap); one run; temperature cannot be set on Opus 5.5, so its answers are not greedy; wait time includes the network round trip to the API.

Data and credits

Test data: CUAD v1 (Hendrycks et al., 2021), the Contract Understanding Atticus Dataset, licensed CC BY 4.0. The model is Qwen3-4B-Instruct-2507 fine-tuned with LoRA on the CUAD training split (separate repository). Model weights are not included here.