docs(rocm): explain the granite-4.0-h-tiny w1 perplexity gap

benchmarks/logit_traces/rocm_gfx1151_a18d3d76/METADATA.txt

@@ -0,0 +1,15 @@
+host: AMD Ryzen AI MAX+ 395 with Radeon 8060S (gfx1151, RDNA 3.5), 96 GiB VRAM carve-out, 31 GiB host RAM
+os: Debian GNU/Linux 13 (trixie), kernel 6.18.12+deb13-amd64
+rocm: 10.0.0 packages (HIP 7.15.26333), ROCM_PATH=/opt/rocm
+mlxcel_commit: a18d3d76 (origin/main, merge of PR #2237)
+mlx_pin: 81ba1c6a0e50a9268b931579c2d4f1158b9aab5a
+rocm_overlay_fork: NripeshN/mlx rocm-support @ 75915908dfe5028335d318b10340313744fd3a8d (src/lib/mlx-cpp/patches-rocm/UPSTREAM), plus the local fixes in LOCAL_FIXES.md (42 items)
+logit_trace_sha256: 687cf66d6867ff43f244a5c6be43034efb029d62a7cbfe075d8bca6d5d78cef5
+build: cargo build --release --features rocm --example logit_trace (default, ssmkernel0, cpubf16 traces); cargo test --profile test-fast --features rocm --lib --no-run with granitemoehybrid_probe_2154.rs attached (f32ref, f32refcpu, f32mix, ops)
+corpus: tests/fixtures/wikitext2_excerpt.txt sha256 16721fc1057d69efc8493718fcad562de9e4554ce05369876cd06dcba431394e
+checkpoint: granite-4.0-h-tiny-4bit@02c8783da0cb171942e30a7dde8acce11fe82e0d (the same local directory as ../rocm_gfx1151_c5fe9a16/)
+args: w1='1 128 8 0'; w1x1024='1 1024 8 0'; w8='8 80 8 512'; w256='256 2 8 0' (CHUNK_TOKENS MAX_CHUNKS TOPK PREFILL)
+tags: 'default' = no override. 'ssmkernel0' = MLXCEL_SSM_KERNEL=0. 'cpubf16' = MLXCEL_DEVICE=cpu, stopped by a 600 s timeout after its first chunk. 'f32ref' = the probe's f32-activation forward on the GPU (embedding output cast to f32, so every op runs in f32 against the bf16-stored weights; logits not rounded to bf16). 'f32refcpu' = the same forward on the CPU stream, first 3 chunks only. 'f32mix-<family>' = the f32 forward with only that family run in bf16 (input cast to bf16, output cast back); 'resid' rounds the residual stream to bf16 after the embedding and after every residual add; 'all' runs every family in bf16. 'ops' = per-stage bf16 output against the f32 stage on the same inputs, first 128 chunks, on the GPU.
+env: LD_LIBRARY_PATH unset; MLXCEL_KEEP_BF16 unset; MLXCEL_FUSED_MOE unset; MLXCEL_DEVICE unset except the cpubf16 trace
+gpu: correctness traces, not timings; every run held the host GPU lock (/tmp/mlxcel-rocm-gpu-guard.lock)
+metal_reference: ../metal_m5_d1128266/ (w1, w8, w256, 128 / 80 / 2 chunks); no Metal host was available to trace 1024 chunks

benchmarks/logit_traces/rocm_gfx1151_a18d3d76/README.md

@@ -0,0 +1,49 @@
+# granite-4.0-h-tiny `w1` against an f32 reference (lablup/mlxcel#2154)
+
+Traces and per-op numbers behind [the granite `w1` section of the 2026-09-30 correctness page](../../../docs/benchmark_results/rocm-correctness-gfx1151-2026-09-30.md#the-granite-w1-perplexity-gap). They answer one question: is the +4.345% `w1` perplexity of ROCm against Metal (`../rocm_gfx1151_c5fe9a16/` against `../metal_m5_d1128266/`) a ROCm defect. `METADATA.txt` has the host, commit, binary hash, checkpoint and the meaning of every tag; `RUNS.txt` has every run; `SHA256SUMS` covers every file here.
+
+## What is here
+
+| File (prefix `rocm_gfx1151_a18d3d76_granite-4.0-h-tiny_`) | What it is |
+|---|---|
+| `default_w1.tsv`, `default_w8.tsv`, `default_w256.tsv` | `logit_trace` at `a18d3d76`, the widths of `../rocm_gfx1151_c5fe9a16/`. The `w1` data rows are byte-identical to that directory's; `w8` and `w256` are not (see the page) |
+| `ssmkernel0_w1.tsv` | `w1` with `MLXCEL_SSM_KERNEL=0`; byte-identical data rows to `default_w1.tsv`, because a `w1` chunk never has SSM state, so neither tag reaches the HIP SSM update kernel |
+| `default_w1x1024.tsv` | `w1` over 1024 chunks; its first 128 rows are byte-identical to `default_w1.tsv` |
+| `f32ref_w1x1024.tsv` | the reference: the same 1024 single-token forwards with f32 activations on the GPU (weights as stored, logits not rounded to bf16) |
+| `f32refcpu_w1x3.tsv` | the f32 forward on the CPU stream, 3 chunks; it matches `f32ref` to 0.0007 nats per position, so the f32 reference does not depend on the device |
+| `cpubf16_w1x1.tsv` | `MLXCEL_DEVICE=cpu logit_trace`, one chunk before a 600 s timeout. Not usable as a reference: its top-1 logit is 14.50 where the GPU gives 18.875 and f32 gives 18.63 |
+| `f32mix-<family>_w1x1024.tsv` | the f32 forward with one op family in bf16: `resid` (the residual stream rounded to bf16 after the embedding and every residual add), `norm`, `mamba`, `attn`, `moe`, `shared`, `head`, and `all` |
+| `ops_bf16_vs_f32.tsv` | per stage, first 128 chunks: relative L2 error of the GPU bf16 stage output against the f32 stage on the same inputs (mean and max over tokens and layers), and the largest absolute error. The last line counts MoE routing sets that differ between bf16 and f32 router logits |
+| `granitemoehybrid_probe_2154.rs` | the probe that wrote the `f32*` and `ops` files |
+
+## Reproducing
+
+```bash
+export OPENSSL_INCLUDE_DIR=/usr/include OPENSSL_LIB_DIR=/usr/lib/x86_64-linux-gnu
+cargo build --release --features rocm --example logit_trace
+T=./target/release/examples/logit_trace; M=models/mlx/granite-4.0-h-tiny-4bit; C=tests/fixtures/wikitext2_excerpt.txt
+$T $M $C 1 128 8 0 > default_w1.tsv
+MLXCEL_SSM_KERNEL=0 $T $M $C 1 128 8 0 > ssmkernel0_w1.tsv
+$T $M $C 1 1024 8 0 > default_w1x1024.tsv
+```
+
+The probe reads private fields of `GraniteMoeHybridModel`, so it has to be compiled as a child module of `src/models/granitemoehybrid.rs`. Copy it to `src/models/granitemoehybrid_probe_2154.rs`, append
+
+```rust
+#[cfg(test)]
+#[path = "granitemoehybrid_probe_2154.rs"]
+mod probe_2154;
+```
+
+to `src/models/granitemoehybrid.rs`, and run it (one GPU process; take the host GPU lock on a shared host):
+
+```bash
+cargo test --profile test-fast --features rocm --lib --no-run
+B=$(ls -t target/test-fast/deps/mlxcel-* | grep -v '\.d$' | head -1)
+export PROBE_MODEL=$M PROBE_CORPUS=$C PROBE_OUT=out
+PROBE_N=1024 PROBE_ARMS=f32gpu $B granitemoehybrid::probe_2154 --ignored --nocapture
+PROBE_N=1024 PROBE_OPS_N=128 PROBE_ARMS=ops $B granitemoehybrid::probe_2154 --ignored --nocapture
+PROBE_N=1024 PROBE_ARMS=none PROBE_MIX='all;resid;norm;mamba;attn;moe;shared;head' $B granitemoehybrid::probe_2154 --ignored --nocapture
+```
+
+The arms `f32cpu` and `bf16cpu` run on the CPU stream at roughly 400 s per chunk on this host, which is why only three `f32cpu` chunks exist.

benchmarks/logit_traces/rocm_gfx1151_a18d3d76/RUNS.txt

@@ -0,0 +1,10 @@
+rocm_gfx1151_a18d3d76_granite-4.0-h-tiny_default_w1                 exit=0 rows=135 elapsed=5s
+rocm_gfx1151_a18d3d76_granite-4.0-h-tiny_ssmkernel0_w1              exit=0 rows=135 elapsed=4s
+rocm_gfx1151_a18d3d76_granite-4.0-h-tiny_default_w1x1024            exit=0 rows=1031 elapsed=20s
+rocm_gfx1151_a18d3d76_granite-4.0-h-tiny_default_w8                 exit=0 rows=647 elapsed=50s
+rocm_gfx1151_a18d3d76_granite-4.0-h-tiny_default_w256               exit=0 rows=519 elapsed=2s
+rocm_gfx1151_a18d3d76_granite-4.0-h-tiny_f32ref_w1x1024             probe PROBE_ARMS=f32gpu,bf16gpu PROBE_N=1024 exit=0 elapsed=265s (both arms)
+rocm_gfx1151_a18d3d76_granite-4.0-h-tiny_f32refcpu_w1x3             probe PROBE_ARMS=f32cpu, stopped by hand after 3 chunks in about 20 minutes
+rocm_gfx1151_a18d3d76_granite-4.0-h-tiny_cpubf16_w1x1               MLXCEL_DEVICE=cpu logit_trace 1 2 8 0, exit=124 (timeout 600s) after chunk 0
+rocm_gfx1151_a18d3d76_granite-4.0-h-tiny_ops_bf16_vs_f32            probe PROBE_ARMS=ops PROBE_OPS_N=128 exit=0 elapsed=261s
+rocm_gfx1151_a18d3d76_granite-4.0-h-tiny_f32mix-*_w1x1024           probe PROBE_MIX='all;resid;norm;mamba;attn;moe;shared;head' PROBE_N=1024 exit=0 elapsed=817s (all eight arms)

benchmarks/logit_traces/rocm_gfx1151_a18d3d76/SHA256SUMS

@@ -0,0 +1,21 @@
+a5d05ae3fe075de2009319e1e378188b68442a7d3586fe9f46c98eb1e01eae92  granitemoehybrid_probe_2154.rs
+5a1fc740ee317045231dc2ff96bfb80e1c6674f66c3b9e84a853ce54cc648795  METADATA.txt
+d0ba50dc3b5a8f22dd5c2b5ba953113e1e8da02ffb1792494de54b269d0b5395  README.md
+9b97dbdfe73388b45949e526dfbea412cf21c48dfa299522fb2144d3402c8fad  rocm_gfx1151_a18d3d76_granite-4.0-h-tiny_cpubf16_w1x1.tsv
+a6caad4d60d0bca640d0b39065ade10f1e412b36f8204fae56510ad9e3c9dbbe  rocm_gfx1151_a18d3d76_granite-4.0-h-tiny_default_w1.tsv
+3a280120835520f7577ae9981c47e3386e8e0a7856de4b1e9e7e834b2c4799e7  rocm_gfx1151_a18d3d76_granite-4.0-h-tiny_default_w1x1024.tsv
+7497fa2557a376ffb39a066561c23b8ee3505945d6291e9abc20768d62845280  rocm_gfx1151_a18d3d76_granite-4.0-h-tiny_default_w256.tsv
+1b1937d4ae8e38af6b01b69d6fe408f48b8f75c7c06c778517f04d07046bb40b  rocm_gfx1151_a18d3d76_granite-4.0-h-tiny_default_w8.tsv
+31e6741075ae8a2f749dadd3b36f0a38cd2df35036caf328e2edaea54c4864b5  rocm_gfx1151_a18d3d76_granite-4.0-h-tiny_f32mix-all_w1x1024.tsv
+f53aa01016ebcb12a1fbca7454764a3545009c9c7cf74ff71e4bcd6174d29f2f  rocm_gfx1151_a18d3d76_granite-4.0-h-tiny_f32mix-attn_w1x1024.tsv
+546a574370a6cc8444d781b070ae86ea8e4bf782a59c3a28b709c9ddb4cdaa88  rocm_gfx1151_a18d3d76_granite-4.0-h-tiny_f32mix-head_w1x1024.tsv
+16312ee9e4417fe7eb2637310a1f428eb90a6bbb4ac0d961ddb05ba7797677de  rocm_gfx1151_a18d3d76_granite-4.0-h-tiny_f32mix-mamba_w1x1024.tsv
+932e6d2ddaa011a25febd1c361d0db6ed21d46be415df9967c016610af0e76f0  rocm_gfx1151_a18d3d76_granite-4.0-h-tiny_f32mix-moe_w1x1024.tsv
+ccefa3135cb4fdd879c0b0b2d7a6610a549e48340997f16c4b5e03a3373d3cb4  rocm_gfx1151_a18d3d76_granite-4.0-h-tiny_f32mix-norm_w1x1024.tsv
+56be35c38331117edbee469850f3faefc02bb20d0c323c67900e4293feffbaa7  rocm_gfx1151_a18d3d76_granite-4.0-h-tiny_f32mix-resid_w1x1024.tsv
+a84f04ddb23995e12ef4cead2c5a1364d4b00530547e13ddba85a5fe3a9e5159  rocm_gfx1151_a18d3d76_granite-4.0-h-tiny_f32mix-shared_w1x1024.tsv
+f8ca0eadd4e31a7224e818debe5f9f77f6d6b9f900c8170efdc428bb5f40175b  rocm_gfx1151_a18d3d76_granite-4.0-h-tiny_f32refcpu_w1x3.tsv
+f1bdf0229f78179899658aea27b7ca9c2d2d97f65e303e7a3f700e1bbc9a385b  rocm_gfx1151_a18d3d76_granite-4.0-h-tiny_f32ref_w1x1024.tsv
+c4b436fa37793aaf136d538724c3fe2f6dab743a9754d6d3306ad7f964be4e66  rocm_gfx1151_a18d3d76_granite-4.0-h-tiny_ops_bf16_vs_f32.tsv
+a6caad4d60d0bca640d0b39065ade10f1e412b36f8204fae56510ad9e3c9dbbe  rocm_gfx1151_a18d3d76_granite-4.0-h-tiny_ssmkernel0_w1.tsv
+9b1675865c0cf9021f80165629a1650b7fbbbbcd38d2b106a8ee76ef162c548e  RUNS.txt