test(rocm): complete the #1809 matrix rows with Metal M5 traces

[inureyes]

Part of #1809

Brings in the Apple M5 Max Metal traces for the five checkpoints PR #2059 traced on ROCm (commit 9830bd6 from data/issue-1809-metal-traces, cherry-picked with its authorship) and completes every row of docs/benchmark_results/rocm-correctness-gfx1151-2026-09-30.md against them, Metal as the reference, with the 2026-09-12 criterion (zero mismatches on positions decided at --decided 2.0). It also adds two w1ctx512 rows that cover Metal's fused SSM update kernel (commits d2043c9 and c570666 from the same branch for the Metal half, plus new ROCm traces).

Result

Pair	Decided mismatches	Verdict
qwen2.5-7b-instruct w1 / w8 / w256	0/19, 0/274, 0/204	pass
gemma-3-4b-it w1 / w8 / w256 / w8ctx1536	0/2, 0/320, 0/204, 0/189	pass
granite-4.0-h-tiny w1 / w8 / w256	0/112, 0/260, 0/208	pass
nemotron-3-nano-30b-a3b w1	0/0	inconclusive (no decided position)
nemotron-3-nano-30b-a3b w8 / w256	0/287, 0/216	pass
qwen2.5-vl-3b-instruct w1 / w8 / w256	0/49, 0/198, 0/154	pass
granite-4.0-h-tiny w1ctx512	0/60	pass
nemotron-3-nano-30b-a3b w1ctx512	0/71	pass

No pair fails. Over the first sixteen pairs top-1 disagrees at 155 of 6720 positions, 0 of 2696 decided; the largest reference gap at a disagreement is 1.250 (Nemotron-H w256, rank 2). The doc also has the logit delta percentiles, perplexity deltas, a 0.5/1.0/2.0 threshold sweep and the rank distribution.

The w1ctx512 rows

The earlier caveat "w1: Metal runs the fused SSM update kernel, ROCm runs the SSD graph" was wrong. The fused step needs seq_len == 1, ssm_kernel_available() and an existing ssm_state, and logit_trace builds fresh caches per chunk, so w1 without prefill runs the graph on both backends: w1 is graph against graph. w1ctx512 (1 128 8 512, MLXCEL_TRACE_START_TOKEN=512) prefills 512 tokens before each single-token forward, so Metal runs ssm_update_kernel (inside fused_mamba2_forward for Nemotron-H) and ROCm runs the SSD graph with state.

• Both halves are at 3c9edea (metal_m5_3c9edea0/, rocm_gfx1151_3c9edea0/), a later commit than the other sixteen pairs. The ROCm binary (sha256 00686315...) was built on this branch at 8e65051, which differs from 3c9edea only in data and docs. The Metal d112826 w1ctx512 traces have byte-identical data rows.
• granite: top-1 4/128, 0/60 decided, largest gap 0.125, perplexity 7.090 Metal vs 7.234 ROCm (+2.0%, about 1 standard error per position). nemotron: 7/128, 0/71, largest gap 0.250, 7.489 vs 7.489. Both are zero at 0.5 and 1.0 too.
• The doc lists the ROCm fixes merged between the two commits and where they sit relative to this path (fix(rocm): stop SliceUpdate donating a still-referenced source #2070, fix(build/rocm): rebuild HIP objects when an included header changes #2076, fix(rocm): run CPU-stream BLAS single-threaded over fine-grained memory #2079, test(rocm): prove mxfp8 end to end, incl. FP8 block checkpoints #2071). Apple has no runtime switch to force the graph path, so fused vs graph could not be A/B tested on Metal.

Not like-for-like (stated in the doc)

• The reference is an M5 (generation 17, NAX kernels for multi-token matmul and prefill attention), not the M1 Ultra of the first run, so the two runs' bounds are against different Metal arithmetic.
• Nemotron-H compares Metal fused_moe_forward with ROCm forward_nonfused at every width.
• The VLM rows trace the language model only.

Findings for follow-up

• src/models/nemotron_h.rs prints five loader lines to stdout, so every Nemotron-H trace (both backends) makes compare_logit_traces.py raise ValueError. Those pairs were compared on copies filtered with grep -v '^\[NemotronH\] '; committed traces are unchanged. The # device header line of newer traces is read as metadata and needs no filter.
• granite w1 perplexity is +4.345% over 128 no-context positions (about 2.5 standard errors per position) with every top-1 in agreement.

Verification

On the gfx1151 host: sha256sum -c SHA256SUMS in all three new or updated trace directories; cargo build --release --features rocm --example logit_trace at 8e65051 and the two w1ctx512 ROCm traces (exit 0, 128 data rows, no NaN), each started with no other process in /sys/class/kfd/kfd/proc; all 18 comparisons run; make verify-versions verify-kernel-dtype-keys verify-kernel-port-dispatch verify-llama-compat verify-rocm-overlay verify-fmt verify-binary-assets pass; cargo test --features rocm --test dead_doc_pointers pass (2 tests).

Not verified: Metal and CUDA are not available on this host. The Metal traces were produced on the M5 Max and are taken on their METADATA.txt, RUNS.txt and checksums. No code on the Metal or CUDA paths changes.