test(benchmark): record the Volta sm_70 baseline and gate sm_70 in CI

[inureyes]

Summary

Phase 0 of epic #1536. Turns the one-off V100 audit that epic rests on into a reproducible committed record with a stated methodology, and adds the minimum build coverage to stop a Volta build rotting silently. No kernel changes. Every number was measured from a build made for this PR (MLX_CUDA_ARCHITECTURES=70 make release-cuda, 96 cubins all sm_70, libmlx.a 155,355,188 bytes) rather than copied, and every disagreement with the audit is reported rather than smoothed over.

The verdict this issue exists to reach

An 8-bit checkpoint decodes faster than a 4-bit one on Volta. 1.886x on a dense controlled pair (gemma-4-12B-it-{4bit,8bit}, whose config.json files differ in exactly two leaves, both bits, with identical parameter counts), and 1.13x on a MoE pair. The 4-bit arm reads 6.17 GB per decode step against the 8-bit arm's 11.65 GB, so the arm moving 1.9x the bytes finishes in 53% of the time; decode on this part is not bandwidth bound.

The mechanism is isolated on the dense pair. Both arms launch identical instance counts for every kernel, so the comparison is of time and not of work: qmv takes 12.8366 s at 4-bit against 5.9911 s at 8-bit over 39,151 instances in each, a delta of 57.05 ms per generated token against a measured wall-clock slope delta of 58.45 ms, so qmv accounts for 97.6% of the entire gap. qmm_naive moves the other way in the same profile (10.0733 s at 4-bit against 11.9215 s at 8-bit over 329 instances each), and that is the control: qmv selects its accumulator on bit width at src/lib/mlx-cpp/patches/mlx/backend/cuda/quantized/qmm/qmv.cu:191 and sm_70 has no bf16 ALU, while qmm_naive accumulates in float regardless of bit width through UniversalFMA<float, Element, Element> in mlx/backend/cuda/device/gemm_sm70.cuh. Where the accumulator is bit-width dependent the 4-bit arm loses by 2.14x; where it is not, it wins as bandwidth predicts. That confirms the hypothesis in #1539 with its counterfactual measured in the same run.

Both dense profiles reconcile against their unprofiled runs (102.4% and 101.6%), which is what makes those absolute times usable.

Headline numbers

Checkpoint	Decode (slope)	Roofline attainment
qwen3.8-27B-4bit	220.33 ms/tok, 4.54 tok/s (spread 0.09%)	65.4 GB/s, 7.27% of 900 GB/s
gemma-4-12B-it-4bit	124.41 ms/tok, 8.04 tok/s	49.6 GB/s, 5.51%
gemma-4-12b-it-8bit	65.96 ms/tok, 15.16 tok/s	176.6 GB/s, 19.63%
gemma-4-26b-a4b-it-4bit	38.61 ms/tok, 25.90 tok/s	MoE, not reported
gemma-4-26b-a4b-it-8bit	34.29 ms/tok, 29.16 tok/s	MoE, not reported

Prefill on qwen3.8-27B-4bit from a five-rung ladder at -n 1: 125.07 ms per prompt token, that is 8.00 tok/s marginal and 0.397 TFLOPS, 2.83% of 14 TFLOPS FP32, with a fixed intercept of 18.41 s. Time to first token is 24.94 s at a 54-token prompt with the model loaded and the cache warm.

What changed

• docs/benchmark_results/volta-sm70-baseline-2026-08-31.md (new, 398 lines). Environment and checkpoint provenance, build provenance including the cubin histogram, the six methodology rules, decode and prefill throughput with roofline attainment for both, the cold-versus-warm PTX cache measurement, the nsys kernel and cuda_api_sum tables, the 4-bit against 8-bit section with the verdict, the three-harness comparison, a full reproduced-or-not table against the audit, the CI decision, the known traps, an empty post-program comparison table keyed to perf(cuda/quant): use float accumulators in qmv below Ampere at bits < 8 #1539 through perf(cuda): CUDA graph instantiation and JIT module load dominate Volta TTFT #1545, and the reproduce commands.
• benchmarks/cuda_v100_2026-08-31.csv (new). Five rows from scripts/bench_decode.sh, in the same 15-column schema the cuda_gb10_* sweeps use.
• .github/workflows/ci.yml. New cuda-sm70-compile job and a new cuda_arch path filter; the header note on CUDA gating updated to say what the three CUDA jobs now cover.
• scripts/bench_decode.sh. A *V100* case in detect_hardware_short so the CSV lands as cuda_v100_<date>.csv beside the GB10 files instead of a 20-character truncation, and MODELS_DIR / BENCHMARKS_DIR made environment-overridable so all mode can sweep a nested owner/name store root. Defaults unchanged, so an unset environment reproduces every historical CSV byte for byte.

Build coverage decision

cuda-sm70-compile runs cargo check --features cuda --all-targets at MLX_CUDA_ARCHITECTURES: "70" on the existing self-hosted CUDA runner, then asserts with cuobjdump --list-elf that the emitted libmlx.a contains sm_70 and nothing else. cargo check still runs the build script, so nvcc compiles MLX even though nothing links, and the cubin assertion is what separates "the environment variable was set" from "nvcc actually emitted pre-Ampere code"; it also catches the 90a auto-detect fallback silently swallowing the architecture list. Path-filtered on cuda_arch (the MLX overlays, the pinned commit, the two build scripts, the workflow) so a Rust-only PR pays nothing, with its own persistent target directory because the architecture list is part of the build-script fingerprint.

A GPU-backed Volta job was considered and declined, and the reasoning is recorded in the job comment and in the document rather than the job being silently omitted: the realistic failure mode this program can introduce is a compile break (a __CUDA_ARCH__ guard not covering cc 7, a CUTLASS type with no pre-Ampere instantiation, a bf16 intrinsic with no sm_70 implementation), no Volta runner exists in this repository's pool, no release artifact targets sm_70, #1537 already turns an architecture mismatch into a named startup error rather than an opaque CUDA load failure, and the one Volta machine that exists is this single-GPU development host where such a job would serialize against the measurements themselves. Adding sm_70 to the release matrix was also considered and declined, since a cold six-architecture build is already about three hours and shipping sm_70 would be a support commitment for a part the current kernels serve badly; revisit once #1539 through #1545 have moved these numbers.

Corrections to the audit

• MoE decode does not route to qmm_naive. A comment on test(benchmark): Volta (sm_70) baseline record and build coverage #1538 inferred it does, from M * B >= 8 reaching GatherQMM. On this build the expert path in decode is mlxcel's own fused custom_kernel_moe_gateup_kernel_cu_* and custom_kernel_moe_down_kernel_cu_*, 3,570 instances each, which is 30 layers over 119 steps; qmm_naive appears with a prefill-sized 326 instances in both arms. The qmv inversion is present in the MoE profile at the same 1.76x the audit found (2.6224 s against 1.4921 s over 28,084 identical instances), but the fused expert kernels move the other way and nearly cancel it, which is why the MoE advantage is 1.13x and not the dense pair's 1.89x.
• TTFT did not reproduce and is recorded as unexplained. The audit reports ~13 s; 24.94 s is measured here at a stated prompt length. A cold PTX cache in the audit is ruled out by direction (cold is slower), and every CUDA API cost in the audit's own profile is higher than this record's at the same call counts. The two are most likely measuring different things, and the audit's prompt length, chat-template use and whether load was included are all unrecorded. This record defines its figure precisely so the next person can compare like with like.
• qmm_naive instance counts are exactly half the audit's on both models (497 against 994, 329 against 658). A factor of exactly two on two independent models is a counting difference, not a performance one, and the conclusion drawn from qmm_naive here is a sign comparison within a single profile, which is unaffected.
• The MoE 8-bit advantage is 12.6% here against 19.9% in the audit. Direction solid over five repetitions per arm, magnitude not: that arm's repeat spread is 13.5%. The MoE numbers are ordinal, not precise.

Methodology the document commits to

Getting any of these wrong changes the answer by more than the effects this epic is trying to measure, and three of them produced a wrong result during this work.

1. Decode rate is a slope over two -n values at a fixed prompt, never tokens / wall_time.
2. Both runs must reach the token budget. A greeting makes every instruct checkpoint here emit EOS after roughly ten tokens, so both -n values return the same generation and the slope collapses; the first attempt at this record did exactly that and produced everything from a division by zero to 193 ms/token on a model whose real figure is 220. The harness now asserts generated_tokens == n before computing a slope.
3. State the PTX cache state. MLX_PTX_CACHE_DIR points at ~/.cache/mlxcel/cuda-ptx/<mlx-commit>, keyed on the pinned commit, so it survives rebuilds. Measured cold against warm on an identical command: 135.13 s against 27.80 s process wall, prefill 76.24 s against 11.24 s, decode 9.95 s against 4.93 s.
4. nsys requires --cuda-graph-trace=node, or roughly all GPU time lands on event_signal_kernel.
5. nsys absolute times count only when they reconcile against an unprofiled wall clock. Reported here: dense pair 102.4% and 101.6% (usable), MoE 4-bit arm 144.0% (not usable for kernel-level conclusions).
6. Percentage shares do not compare across runs. The qmm_naive share for qwen is 70.4% here and 82.0% in the audit while the per-instance times agree, purely because the totals differ.

make bench-model violates rule 1 and reads 2.1x to 9.1x low on this host (qwen3.8-27B-4bit 1.67 tok/s against a true 4.54). On the MoE pair it is not even sign stable: it reports 4-bit ahead of 8-bit (3.28 against 3.21) while both the slope and scripts/bench_decode.sh put 8-bit ahead by 12% and 18%. A Volta record quoting the Makefile harness alone would have published the opposite of this PR's central finding, so the document reports the slope alongside it and says why. scripts/bench_decode.sh, which warms in-process, lands within 2% to 15% of the slope and stays the artifact producer.

Test plan

Host: Tesla V100-PCIE-32GB (compute capability 7.0, sm_70), CUDA 12.9.41, driver 575.51.03, x86_64, single exclusive GPU. Every run verified with nvidia-smi --query-compute-apps empty at its start after one overlap was detected and its results discarded.

• MLX_CUDA_ARCHITECTURES=70 make release-cuda succeeded (25m30s), and cuobjdump --list-elf on the resulting libmlx.a reports 96 cubins, all sm_70, nothing else.
• MLXCEL_TRACE_ARCH=1 ./target/release/mlxcel generate ... reports compute capability 7.0 (sm_70); compiled for [70]; coverage: cubin, so the archive and the runtime agree.
• Decode slopes on all five checkpoints, three repetitions for the dense models and five for the noisier MoE pair, each with generated_tokens == n asserted. Dense spreads 0.09% to 3.28%.
• Prefill ladder on qwen3.8-27B-4bit at five prompt lengths (54, 161, 365, 671, 1283 tokens), two repetitions each, fitted by least squares.
• Cold against warm PTX cache measured on an identical command, cache moved aside and restored intact (22M before, 22M after).
• Five nsys captures with --cuda-graph-trace=node, each reconciled against its unprofiled counterpart, with the ratio reported including the MoE arm that fails the check.
• MODELS_DIR=./models/mlx-community ./scripts/bench_decode.sh all produced the committed CSV; detect_hardware_short verified to return v100 for this host, matching the committed filename.
• make bench-model on all five checkpoints for the harness comparison, followed by make bench-clean; benchmark_results.log is absent and the tree is clean.
• bash -n scripts/bench_decode.sh and a YAML parse of .github/workflows/ci.yml confirming the new job and the new changes output are well formed.
• python3 validate_body.py on the document, this body, and the commit message.

Deferred to GB10

None of the following was verified, because this machine has exactly one GPU and it is a Tesla V100; no Ampere-or-later device exists here. They belong to the ## GB10 (sm_121) continuation section of epic #1536, which lists for this issue: "Nothing device-specific; confirm the baseline doc's methodology section still describes the GB10 sweep correctly."

• The methodology section against a GB10 sweep. The document's rules were derived on sm_70. Rule 1 (slope over single-run) matters far less on GB10, where the fixed cost is small, and rule 3 (PTX cache state) has not been checked there at all. Someone on GB10 should confirm the section still describes that sweep correctly, and amend it where a rule is Volta-specific rather than general.
• cuda-sm70-compile has never executed. GitHub Actions cannot be run from here. The job's cargo check --features cuda --all-targets at sm_70 and its cuobjdump assertion were verified only by the equivalent local build, whose archive the assertion's condition was written against; the job itself, its path filter, its concurrency group and its target-directory step are unexercised. Its first real run will be this PR, on the cuda_arch filter that .github/workflows/ci.yml matches.
• The bench_decode.sh change on the GB10 sweep path. MODELS_DIR and BENCHMARKS_DIR defaults are unchanged and the new *V100* case cannot match a GB10 host, so the GB10 all sweep is unaffected by construction, but it was not re-run to confirm byte-identical output.
• Whether the graph-construction cost is Volta-specific. The document records graph construction plus module load at 3.3 s of a 24-token qwen run against 5.3 s of decode kernel time. Whether that ratio is a Volta problem or a general one is the open question perf(cuda): CUDA graph instantiation and JIT module load dominate Volta TTFT #1545 raises and needs a GB10 comparison to answer.

Closes #1538