test(rocm): prove mxfp8 end to end, incl. FP8 block checkpoints

[inureyes]

Summary

• Committed numeric tests for the two mxfp8 matmuls mlxcel runs, through the production layers: SwitchLinear::forward (gather_qmm) and UnifiedLinear::forward (quantized_matmul), against a host-decoded f64 reference (src/models/switch_layers_mxfp_tests.rs). The gather cases hit every branch a block-float mode reaches in the ROCm GatherQMM::eval_gpu: all specialised paths there are affine-only, so mxfp4/mxfp8 reach only the item-10 gather_qmv_kernel<T, uint8_t, BITS, 32, false> launch. The cases cover bf16/f16/f32, 8 and 4 bits, unsorted and sorted-rhs schedules (activation stride K and 0), and M = 1 and 4. This is the default path. The opt-in expert-batched kernel (item 9) is affine-only and unreachable. A reduced matrix also runs on the CPU device (both modes, unsorted and sorted gather, multi-row, dense M = 1 and 4, bf16 and f16, 64-wide output), which takes 2.5 s instead of the 38 s the full matrix needed on MLX's scalar CPU fp_qmm_t.
• End-to-end run of a vendor FP8 block checkpoint on ROCm against the CPU device, with results in docs/benchmark_results/rocm-fp8-block-gfx1151-2026-09-30.md and traces in benchmarks/logit_traces/fp8_block_gfx1151/. Getting that reference took two fixes. examples/logit_trace now honours MLXCEL_DEVICE. The ROCm overlay's ScaledDotProductAttention::use_fallback now takes the fallback on a CPU stream (LOCAL_FIXES.md item 23). Before it, MLXCEL_DEVICE=cpu aborted every attention model with NYI on a ROCm build. tests/cpu_device_sdpa.rs guards it.
• Step 3 decision: load-time requantization stays on the default device. GPU-quantized and CPU-quantized weights (both computed on the GPU) disagree at 0 of 346 decided positions, and CPU quantization takes the 0.8B load from under a second to about 4 minutes. This is documented in fp8_block.rs. No new round-trip test was needed because the quantize stream did not change. The hardware.rs evidence comment now cites these results.

Verification

All on gfx1151:

• cargo test --features rocm --lib models::switch_layers::mxfp_tests models::fp8_block -- --test-threads=1: 16 passed, including the existing round trip. Worst relative L2 error on ROCm: 1.8e-3 (bf16), 2.2e-4 (f16), 2.9e-7 (f32). On the CPU device, reduced matrix: 8.2e-3 (bf16), 1.0e-3 (f16); the earlier full-matrix run gave 7.9e-3, 1.0e-3, 1.2e-7. Bounds are 2e-2, 4e-3, 2e-3 (the f32 bound covers CUDA's default TF32 grouped GEMM, which the sorted prefill case reaches on sm80 and later).
• With LOCAL_FIXES item 10 reverted and the overlay rebuilt, both gather tests failed with non-finite output. With the SDPA guard reverted, cargo test --features rocm --test cpu_device_sdpa aborted with NYI. Both pass restored.
• Checkpoint Lazarus-Ai/ReAligned-Qwen3.5-0.8B-FP8 @ db97e6a7 (dense). CPU vs ROCm at w32 (32 8 8 0): 0/50 decided disagreements, largest gap 0.125. ROCm traces are byte-identical on rerun and on the rebased build.
• mlxcel generate on ROCm: 132 tensors requantized in 565 ms, 56 to 62 tok/s, coherent text ("The capital of France is Paris ... The Eiffel Tower comes to mind immediately. The Louvre Museum is another major landmark.").
• make verify-versions verify-kernel-dtype-keys verify-kernel-port-dispatch verify-llama-compat verify-fmt, cargo test --test dead_doc_pointers, and clippy -D warnings on mlxcel (lib, tests, examples) and mlxcel-core (lib, tests): pass.

Not verified: Metal and CUDA are not available on this host. On those paths the change touches examples/logit_trace (now calls initialize_runtime_checked, which also reads MLXCEL_DEVICE there and applies the default wired limit and the arch-compatibility refusal the CLI already applies) and adds tests/cpu_device_sdpa.rs and the mxfp tests, which run on every backend. Neither the SDPA edit (ROCm overlay only) nor any library code path changes there. Also not verified: the official Qwen3.5 FP8 MoE and 27B releases (too large for this host and for a CPU reference), and CPU-reference arms at w1, w8 and w256 (the CPU backend needs about 2.7 minutes per token for this checkpoint).

Closes #1807

[inureyes]

Implementation Review Summary

Intent

Prove mxfp8 on ROCm for the MoE gather_qmm path and for vendor FP8 block checkpoints requantized at load, with committed op tests and a checkpoint trace against a reference device.

Findings Addressed

• SortedSharedActivation used 16 slots over 8 experts, so B / E = 2 missed the ROCm sorted-rhs gate (B / E >= 4) and the stride-0 implicit-lhs branch the test doc claims was never run. Now 32 slots (HIGH, 17f069e)
• f32 bound 1e-5 would fail on CUDA sm80+: the sorted prefill case (B >= 8 * E) takes MLX's grouped GEMM, which runs f32 through TF32 by default (MLX_ENABLE_TF32=1). Bound now 2e-3 (HIGH, 17f069e)
• Section 3 of the results doc did not name its reference arm; the quoted 346 decided positions, 0.375 gap and 39/42 rank-2 hold only with rocm_cpuquant as the reference. Now stated, with the command (MEDIUM, 17f069e)
• METADATA.txt gave the MLX pin 81ba1c6a as the mlxcel base (LOW, 17f069e)
• PR body: measured maxima refreshed, and the logit_trace note now says it also applies the default wired limit and the arch refusal on Metal/CUDA (LOW)

Remaining Items

• That the 32-slot case reaches use_sorted_rhs_schedule is shown from the gate in qmm.hip, not by instrumenting the kernel (LOW)

Verification

• All stated requirements implemented
• No placeholder/mock code remaining
• Integrated into project code flow
• Project conventions followed
• Existing modules reused where applicable
• No unintended structural changes
• Tests pass (gfx1151: models::switch_layers::mxfp_tests 5/5, models::fp8_block 11/11, cpu_device_sdpa, dead_doc_pointers, clippy -D warnings on the mlxcel lib and tests, verify-fmt; compare_logit_traces.py on the committed TSVs reproduces every number in the doc and the PR body)