feat: add ADR 0007, CLI/server parity harness and B=1 decode bench

[inureyes]

Summary

Phase 0 of epic #2166. ADR 0007 records the batch-native engine design and the CLI-vs-server defaults table, two tools measure where the two decode paths stand today, and a baseline run on GB10 fills the rows the ADR decides by measurement. No decode behavior changes.

What is in it

• ADR 0007 (docs/adr/0007-unified-batch-native-engine.md, indexed in docs/adr/README.md):
  • It names the engine API (Engine, KvStore over dense and paged storage, PrefillPlan, RowSampler, FinishStep) and maps refactor: one post-sample finish step for every CLI and server decode site #2168 to refactor: run mlxcel generate on the engine and retire CxxGenerator #2176 onto it.
  • It supersedes ADR 0004's KV/scheduler deferral and cites the llama.cpp design (batch-native llama_decode, one common_sampler, llama-cli as a client of llama-server).
  • Semantic defaults are decided with reasons: repeat_last_n 64, DRY window and the b10621 breakers, loop detection in the single finish step, server sampler defaults exposed to every client, greedy-by-default CLI clients, the per-slot max_tokens rule, per-sequence seeds, prompt cache on for chat clients.
  • Speed defaults are decided by measurement, the faster option winning (maintainer decision on test: ADR 0007 plus CLI/server parity and B=1 decode benchmark harness #2167): prefill chunk 2048, single-sequence KV storage dense, regression threshold 1.0 percent.
• make engine-parity (mlxcel-engine-parity) runs one prompt and SamplingConfig through CxxGenerator and through the in-process server scheduler at B=1, dense and paged.
  • It also runs a prompt-cache miss vs hit, recording each side's prefill partition.
  • Per pair it prints identical or the first divergent index and both token ids, plus the paged kernel launch count on the paged row.
  • --expect-identical exits 1 on divergence, and the tool refuses to run without MLXCEL_SDPA_DETERMINISTIC=1.
• make bench-engine (mlxcel-bench-engine) times bench_decode's CxxGenerator call next to the server worker driven in-process.
  • HTTP and tokenization are outside the timed region.
  • It runs at 256 and 8192-token prompts, with --prefill-chunk (both paths) and --decode-storage (server).
  • scripts/engine_bench_rounds.py runs interleaved rounds with a rotating order, null arms, a quiet-host gate and a per-run timeout.
• src/server/engine_probe builds the server the way start_server does. The prompt-cache store setup moved verbatim into resolve_prompt_cache_store so both share it, and the long-prompt corpus moved out of bench_decode so both benches synthesize the same prompt.

Baseline (GB10, docs/benchmark_results/unified-engine-baseline-gb10-2026-10-07.md)

• Server single-stream decode today is 8 to 23 percent below the CLI (Qwen3-1.7B 0.77x at 256 tokens, 0.92x at 8192; Llama-3.2-1B 0.90x at both). Almost all of it is paged storage at B=1: with dense storage the server path is within 1.5 percent of the CLI.
• Prefill chunk: 512 raised TTFT at 8192 tokens by 8.4 to 29.4 percent against 2048, on both paths and both models. Every null range was within ±1.9 percent, and decode did not move.
• Storage: paged decode at B=1 was 6.5 to 23.0 percent slower than dense in every cell, with the paged kernel confirmed running. Every null range was within ±1.1 percent.
• Parity: under MLXCEL_SDPA_DETERMINISTIC=1, CLI, server-dense and server-paged are identical over 400 tokens for both models, greedy and seeded with penalties and DRY. Only the prompt-cache hit diverges: on Qwen3 greedy at token 0, and seeded at token 11 on Qwen3 and token 25 on Llama. That divergence is refactor: one prefill planner and one pad-trim hook for CLI and server #2170's first target.

Changes during review

• Paged attention reporting: a "paged" B=1 run could actually be dense attention for families that own their KV state. The tools now report paged kernel launches and flag a paged row that ran dense.
• CUDA graph setup: the in-process server now applies the same main-thread CUDA-graph disable start_server applies.
• Rounds script: arm order now rotates. A per-run timeout and a reserved -null suffix guard were added.
• Prompt validation: too-short prompts are rejected up front.
• Tests: unit tests added for report formatting, argument validation and the rounds script.

Test plan

• cargo test --profile test-fast --features cuda --lib server::engine_probe (14). Bin tests: engine_parity 18, bench_engine 11. tests/test_engine_bench_rounds.py 12.
• cargo clippy --profile test-fast --features cuda --lib --tests --bins -- -D warnings and cargo fmt --check
• Real checkpoints, functional:
  • make engine-parity on qwen3-0.6b, qwen3-1.7b, llama-3.2-1b and gemma-3-1b (the last confirming the dense-attention annotation).
  • The bench and the rounds script exercised end to end.
• Baseline and A/B measurements run once on a quiet host under gpu-lock, raw records committed under docs/benchmark_results/data/unified-engine-baseline-gb10-2026-10-07/

Closes #2167

[inureyes]

Implementation Review Summary

Intent

Phase 0 of #2166: ADR 0007 with the CLI-vs-server defaults table, a CLI/server parity harness and a B=1 decode bench that drive the real CxxGenerator and BatchScheduler, with no decode behavior change.

Findings Addressed

• A paged (c) run could not be told apart from dense attention. Model-owned KV families (Gemma 3, Llama 4, Qwen 3.5) resolve to paged storage but decode a lone sequence through their dense caches. On gemma-3-1b-it-4bit the "Paged" row launched zero paged kernels. The run now records paged_decode_launches from mlxcel_core::cache::paged_batch_decode_stats, the harness prints it on the (c) row, and the bench writes it per server row (f2d1f79) (MEDIUM)
• ServerEngine::start skipped the main-thread CUDA graph hoist start_server runs for Gemma 4 (bug(gemma4/cuda): decode collapses to <pad> after 1-3 tokens (single-query maskless attention + quantized GEMV); mlx_lm decodes coherently #688) (f2d1f79) (MEDIUM)
• engine_bench_rounds.py ran the arms in the same order every round. The order now rotates per round (f2d1f79) (MEDIUM)
• ADR 0007 corrections: the auto-to-paged condition, the paged-kernel caveat for the storage A/B, and seed scope (per-row reseed happens at the first token only) (f2d1f79) (LOW)

Remaining Items

• --expect-identical exits 1 both on divergence and on errors, including the missing MLXCEL_SDPA_DETERMINISTIC refusal, so a gate cannot tell the two apart (LOW). Left as is because the ADR documents exit 1.
• The prefill partition on the cache rows is predicted from the scheduler's own rule (history_boundary_len), not observed (LOW).

Verification

• All stated requirements implemented
• No placeholder/mock code remaining
• Integrated into project code flow
• Project conventions followed
• Existing modules reused where applicable
• No unintended structural changes
• Tests pass

[inureyes]

Security and performance review

No CRITICAL, HIGH or MEDIUM findings, so nothing was committed.

• resolve_prompt_cache_store is a verbatim move: same position in start_server (after the router early return, before ModelProvider creation), same capacity defaults, hybrid-SSM APC rule and metrics wiring. No validation, auth or limit step is skipped.
• Visibility widening is crate-internal (pub(super) for warmup_model and resolve_prompt_cache_store, pub(crate) for history_boundary_len). engine_probe is referenced only by the two new bins and bench_decode; no route, router or startup path reaches it.
• set_var("MLXCEL_PREFILL_CHUNK") in bench_engine runs before initialize_runtime and before any thread exists, so it is sound. ServerEngine::start calls maybe_disable_cuda_graphs_for_model_for_path on the calling thread, the same posture start_server has.
• No production hot path changed. bench_decode's prompt synthesis and context cap are behavior-identical after the move.
• engine_bench_rounds.py builds argv lists (no shell); arm flags go through shlex.split. Every input is operator-supplied.

LOW, left open:

• src/bin/bench_engine.rs:128: with --path both --prefill-chunk N the env var also reaches the in-process server worker, where the Gemma 4 MTP target prefill reads it (only under speculative serving). --path server does not set it, so those two arms can differ for MTP models.
• engine_bench_rounds.py: an arm the user names X-null collides with the generated null arm for X, and subprocess.run has no timeout, so a hung bench blocks the round.
• engine_parity.rs: an empty or one-token prompt (--no-chat-template -p "") gives a zero-length prime or CLI prompt. The run errors out instead of reporting it.

Checks run: cargo check and clippy -D warnings (--lib --tests --bins, cuda, test-fast) are clean, and cargo test --lib server::engine_probe passes 12 of 12. No GPU runs.