test(cuda): run the deferred CUDA arm for the merged ROCm routing work on a GB10 node #1873

Part of #1801

Problem / Background

The changes listed below are already merged to main. This issue is a verification pass, not a merge gate: it can find a regression that already shipped, and a failure here means opening a fix issue against the specific change, not reverting the set blindly.

The CUDA arm was deferred at merge time because no CUDA host was reachable from the ROCm development machine or from the Metal host, and the self-hosted CUDA CI runner has been down (its jobs sat queued and expired at the 24 hour timeout). Rather than leaving an implied "someone should check CUDA" note in each PR, the whole deferred set is collected here to be run in one pass when a GB10 node is available.

Scope widened on 2026-09-29. The routing work this issue covers grew after it was filed. PRs #2010, #2018, #2026 and #2029 each changed code on the CUDA path, and none was run on CUDA, because the only development host is the gfx1151 machine. PR #2029 says so explicitly in its "Not verified" section ("Metal and CUDA were not run; neither is available on this host"); #2010, #2018 and #2026 have no such section, but each records its validation as gfx1151-only, and #2018 argues CUDA is unaffected by reasoning ("CUDA and Metal are unreachable from every new path, since custom_kernels_available() is true there") rather than by a run. All four are now in scope, and the arm is main at or after d8d34e2b (PR #2029).

What needs the CUDA arm

Procedure

This is written for someone with no context from these PRs, on a GB10 node with a working CUDA toolchain and a clone of this repository.

1. Ask the binary which backend it resolved

The merged work ships its own probe, so the central risk of this whole issue does not have to be inferred from counters. gpu_kernel_backend() in src/lib/mlx-cpp/turbo/gpu_backend.cpp resolves the backend once behind a static, and when MLXCEL_DEBUG_KERNEL_BACKEND is set in the environment (gpu_backend.cpp:73) it prints exactly one line to stderr:

[mlxcel] custom kernel backend: cuda

When the resolved backend has no custom kernel ports it appends (no custom kernel ports; using MLX graph fallbacks). Run the arm binary once with the variable set and read that line:

MLXCEL_DEBUG_KERNEL_BACKEND=1 ./target/release/mlxcel generate --model models/mlx/Meta-Llama-3.1-8B-Instruct-4bit --prompt hi -n 8 2>&1 | grep 'custom kernel backend'

Any run that touches a kernel-availability predicate triggers it, and a short generate does, so no special model or flag is needed. (Corrected 2026-09-29: this used to say fused_norm.cpp and sampling.cpp both call custom_kernels_available(); since PR #2026 they call has_kernel_port, which resolves the same gpu_kernel_backend() at kernel_port.cpp:53-54, so the probe still fires.) Pass condition: the line reads cuda with no suffix. That single line settles the risk the rest of this issue is built around, which is that a new predicate answers wrongly and every fused path quietly falls back to an MLX graph with identical output and only lower throughput. It costs one run, so do it first: if it fails, nothing below is worth running, and the failure itself is the bug report.

2. Build both arms

The baseline is 56d82fbc (fix: refuse byte-identical arms in the A/B output gate (#1867)), the last commit on main before this ROCm series began landing; 4641dc7a (feat(rocm): implement FFT through hipFFT, PR #1861) is the first of the series on main. The arm is main at or after d8d34e2b (PR #2029), so it includes the whole widened set. Name the two binaries target/release/mlxcel.before (baseline) and target/release/mlxcel (arm).

git checkout 56d82fbc
cargo build --release --features cuda --bin mlxcel
/bin/cp target/release/mlxcel target/release/mlxcel.before
git checkout main
cargo build --release --features cuda --bin mlxcel 2>&1 | tee /tmp/build-arm.log

The check that matters is in the second build's log: src/lib/mlxcel-core/cpp/mlx_cxx_kernels.cpp, mlx_cxx_bridge.cpp, and the files under src/lib/mlx-cpp/turbo/ (kernel_port.cpp, paged_attention.cpp, paged_attention_v2.cpp, paged_attention_v2_merge.cpp, sampling.cpp, sampling_rejection.cpp, fused_norm.cpp, fused_rope_append.cpp) must actually recompile. A cargo build that no-ops after a bridge .cpp edit has been observed on this project, and when it happens the arm binary is secretly identical to the baseline, at which point every check below passes for the wrong reason. Grep the log for those filenames and keep the matching lines as evidence.

3. The test gate

This is the strongest of the automated checks, because the tests that exercise these launchers call them directly rather than through a model, so no token floor or default-off gate can quietly skip them.

git checkout 56d82fbc && make verify-test-cuda 2>&1 | tee /tmp/test-before.log
git checkout main && make verify-test-cuda 2>&1 | tee /tmp/test-after.log
grep '^test result:' /tmp/test-before.log > /tmp/result-before.txt
grep '^test result:' /tmp/test-after.log > /tmp/result-after.txt
diff /tmp/result-before.txt /tmp/result-after.txt

Pass condition: the same tests pass in both runs, and nothing that passed before fails after. The groups whose results matter most are paged_v2::*, mla::split_kv, cache::paged_batch_decode and models::bitnet::*, and (added 2026-09-29 for the widened set) sampling_gumbel_tests, sampling_rejection_tests, the sampling_gumbel_kill_switch and sampling_rejection_kill_switch test targets, fused_norm_parity_tests, fused_rope_parity_tests, fused_moe_parity_tests and rms_norm_small_axis_tests. A group that goes from passing to skipped is a failure, not a pass. make verify-test-cuda is the CUDA gate (Makefile:775); there is no separate verify-cuda aggregate target.

(Added 2026-09-29) On the arm, also run the two toolchain-free checkers the widened set added to or relied on, which must pass unchanged on a CUDA host:

make verify-kernel-port-dispatch   # scripts/ci/check_kernel_port_dispatch.py (Makefile:818)
make verify-kernel-dtype-keys      # scripts/ci/check_kernel_dtype_keys.py (Makefile:813)

Pass for the widened set: make verify-test-cuda at its prior baseline (no new failure, no group newly skipped), and the port checker reporting 10 launcher file(s) route through select_kernel_port, 0 exempt, as it does on d8d34e2b.

4. Greedy output equality

./scripts/ab_output_equality.sh \
  --baseline target/release/mlxcel.before \
  --arm target/release/mlxcel \
  --model models/mlx/Qwen3-30B-A3B-4bit \
  --model models/mlx/Meta-Llama-3.1-8B-Instruct-4bit \
  --prompt "Explain how paged attention differs from a contiguous KV cache." \
  -n 64

The script pins --temp 0 and --show-reasoning on both arms, so the comparison covers every generated token and is not a comparison between two samples. It also runs the baseline twice as a self-consistency control and reports INCONCLUSIVE rather than FAIL when the baseline does not reproduce itself. It refuses to start when the two binaries are byte-identical; that refusal is the no-op build from step 2 surfacing, not a fault in the script. Exit status: 0 all pairs equal, 1 at least one pair differs, 2 inconclusive with no differences.

5. Reach the three changed paged bridges

mlxcel generate never reaches any of them: it is single-sequence and dense, and the fused paged path needs a pool-backed cache. Each bridge has its own production route, and they are not the same route, so "run a server" is not a single answer for all three.

So run three things rather than one:

# v1 paged_attention_decode: the only production route is the tuner.
MLXCEL_DEBUG_KERNEL_BACKEND=1 ./target/release/mlxcel tune --op paged-decode-splits

# v2 partial (+ merge on long contexts): the batched server path, not --kv-unified.
MLXCEL_PAGED_ATTENTION_NATIVE=1 MLXCEL_DEBUG_KERNEL_BACKEND=1 ./target/release/mlxcel-server --model models/mlx/Qwen3-30B-A3B-4bit --max-batch-size 4 --decode-storage-backend paged --metrics

# merge_states on its own route: DeepSeek MLA split-KV, reachable from plain generate.
MLXCEL_MLA_SPLIT_KV=1 ./target/release/mlxcel generate --model models/mlx/DeepSeek-V2-Lite-4bit --prompt "..." -n 64 --temp 0

For the server arm, send several concurrent chat completions with long prompts so more than one request decodes at once and each clears the dispatch floor, then repeat the identical sequence against the baseline binary. The workspace test gate in step 3 covers all three bridges directly and is the check to trust if a route above cannot be driven on this node. scripts/paged_decode_counter_ab.sh (PR #1881) encodes the server arm and the four traps from the 2026-09-14 comment below, and is backend-agnostic.

The paged arm's pass condition is the decode counters, not output equality

Output equality cannot see this failure. The gather fallback is output-identical to the fused path by construction, so an arm that silently stopped launching the fused kernel scores a clean PASS in step 4. There is an in-tree instrument for exactly this question, so use it rather than inferring.

paged_batch_decode_stats() (src/lib/mlxcel-core/src/cache/paged_batch_decode.rs:213) returns the process-wide counters v2_launches, gather_fallbacks, declines and cascade_launches (plus the cascade token / member sums and cascade_failures). The scheduler publishes them every tick: run_loop.rs:415 calls update_paged_decode_gauges(mlxcel_core::cache::paged_batch_decode_stats()), and /metrics renders three of them (src/server/routes/metrics.rs:225-229), which is why the server command above carries --metrics (the route is mounted unconditionally but the handler answers 501 without the flag, src/server/app.rs:494-499, was :475):

curl -s localhost:8080/metrics | grep mlxcel_paged_decode_launches_total
# mlxcel_paged_decode_launches_total{path="fused_v2"} N
# mlxcel_paged_decode_launches_total{path="gather"}   N
# mlxcel_paged_decode_launches_total{path="cascade"}  N

These are values rather than log lines on purpose, and the doc comment on PagedBlockPool::paged_decode_batched (src/lib/mlxcel-core/src/cache/paged.rs:1973-1976, was :1948-1951) records why: the outcome "is a value rather than a log line because the first cut of this used tracing::debug! and a whole production benchmark sweep then compared gather against gather without anything saying so." That is this issue's failure mode with a different cause, which is what makes the counters the pass condition here rather than a nice-to-have.

Pass condition for the paged server arm, read after the identical request sequence on each arm:

• v2_launches (path="fused_v2") is non-zero on both arms and moves together across the two runs. Zero on either arm means the fused path never ran and everything else measured gather against gather.
• gather_fallbacks (path="gather") does not rise on the arm alone. This is the counter the regression this PR could introduce actually moves: PR refactor(core): route custom kernels by GPU backend kind #1877 (earlier cited here as PR refactor(core): route custom kernels by GPU backend kind #1869, which was closed unmerged) added an early return at the top of PagedBlockPool::paged_decode_batched (src/lib/mlxcel-core/src/cache/paged.rs:2006-2013, was :1981-1988) that returns Ok((None, PagedDecodeOutcome::NotServable("the GPU backend has no fused paged-attention kernel port"))) when custom_kernels_available() is false, and paged_batch_decode_attention folds an Ok((None, _)) into gather_fallbacks (paged_batch_decode.rs:375), not into declines.
• declines (not published to /metrics; read it from a test or a debug build) is the pre-pool reject count, bumped only when batch_is_servable or the single-token shape check rejects the batch before the pool is touched (paged_batch_decode.rs:328). It should not differ between arms either, but the new early return does not touch it, so it is corroboration and not the primary signal.

The server log corroborates the number with a reason. report_once (paged_batch_decode.rs:152) announces the first occurrence of each distinct outcome kind through tracing::info!, deliberately at info rather than debug, so the line paged decode v2: gather: batch not servable (the GPU backend has no fused paged-attention kernel port) appears without anyone enabling a debug filter. Read it as corroboration only: the flags are keyed by outcome kind, and NotServable is one kind (index 8, src/lib/mlxcel-core/src/paged_v2/outcome.rs:130), so an ordinary batched prefill declining with NotServable("not a single-token decode step") earlier in the run can claim the slot and suppress the kernel-port line. The counter cannot be pre-empted that way.

The two instruments answer different questions and both are needed. MLXCEL_DEBUG_KERNEL_BACKEND from step 1 says what the backend predicate resolved to; the counters say whether the fused path actually ran. A build can print cuda and still serve every step from gather (dispatch floor, multi-slab layer, MLXCEL_PAGED_ATTENTION_NATIVE kill switch), and the counters alone cannot say which of those it was. Post both.

Step 3 already exercises the same instrument at test level: tests/cascade_decode_dispatch.rs:185 reads paged_batch_decode_stats() and asserts it agrees with the returned outcome, on the grounds that otherwise the /metrics view of which kernel ran is decorative.

Two corrections to an earlier revision of this issue, recorded so they are not reintroduced. It said the server arm with --kv-unified and MLXCEL_PAGED_ATTENTION_NATIVE=1 exercised paged_attention_decode; it does not, because that bridge sits behind the library-only pooled entry point and the server never calls it. And --kv-unified is not what enables the paged pool: it only collapses the parallel context slots to one for sizing (effective_parallel_context_slots, src/server/startup.rs:838, was :788) and the batch scheduler does not read it at all.

Acceptance Criteria

• The [mlxcel] custom kernel backend: line from step 1 is posted and reads cuda with no fallback suffix.
• The two test result: summaries from step 3 are posted and show no new failures and no group that went from passing to skipped.
• The A/B script's exit status and its per-pair verdict lines from step 4 are posted.
• The build log evidence from step 2 is posted, showing mlx_cxx_kernels.cpp and the src/lib/mlx-cpp/turbo/ sources actually recompiling in the arm build.
• Each of the three routes in step 5 is either driven and reported, or explicitly recorded as not runnable on this node with the reason.
• The mlxcel_paged_decode_launches_total fused / gather / cascade split from step 5 is posted for both arms, showing fused_v2 non-zero on both and gather not rising on the arm alone.
• Any difference found is filed as its own issue, naming the arm it appeared on and the exact command that produced it.
• (Added 2026-09-29) The arm is built from main at or after d8d34e2b, so PRs fix(rocm): refuse instead of aborting at the sampler entry points #2010, fix(rocm): guard the four remaining custom-kernel launchers #2018, refactor(core): choose custom-kernel ports through one helper #2026 and refactor: finish the kernel port standardization #2029 are covered, and the posted make verify-test-cuda result shows the groups step 3 names for them passing, not skipped.
• (Added 2026-09-29) make verify-kernel-port-dispatch and make verify-kernel-dtype-keys output from the GB10 node is posted, both passing.
• (Added 2026-09-29) The MLXCEL_FUSED_MOE_RELU2 Nemotron-H path is run once on CUDA if a checkpoint is available, and the outcome (panic at nemotron_h.rs:1424, or not) is recorded, with a follow-up issue if it panics.

Note

A plain "all equal" is not sufficient on its own. It needs the step 2 build evidence and the step 1 backend line beside it, because the two most likely failure modes in this set both produce exactly an "all equal" result: a no-op rebuild that leaves both arms pointing at the same code, and a predicate that answers wrongly so every fused path silently takes an MLX graph fallback with identical output and lower throughput. For the paged server arm it also needs the step 5 counter split, which is the only one of the three that observes the fused path actually running.

Refresh log

2026-09-29

• Widened the scope to PRs fix(rocm): refuse instead of aborting at the sampler entry points #2010, fix(rocm): guard the four remaining custom-kernel launchers #2018, refactor(core): choose custom-kernel ports through one helper #2026 and refactor: finish the kernel port standardization #2029, one row each for what it changed on the CUDA path; the arm is now main at or after d8d34e2b. Only PR refactor: finish the kernel port standardization #2029 states in a "Not verified" section that CUDA was not run; the other three record gfx1151-only validation.
• Corrected the landing PR for refactor(core): route custom kernels by GPU backend kind instead of treating every non-Metal GPU as CUDA #1803 from PR refactor(core): route custom kernels by GPU backend kind #1869 (closed unmerged) to PR refactor(core): route custom kernels by GPU backend kind #1877 (d492c81b), in the table and in the gather_fallbacks bullet.
• Corrected moved line references after re-verifying each at d8d34e2b: paged.rs (:1982, :1973-1976, :2006-2013, :2195), layers.rs:5704, launch.rs:143, cli_input.rs:1217, llama3.rs:1608, qwen3.rs:1179, app.rs:494-499, startup.rs:838. Step 1 now says the probe fires through has_kernel_port, not custom_kernels_available().
• Added the widened set's test groups, the two checkers, and the pass definition (make verify-test-cuda at its prior baseline plus the port checker passing), and a traced, not run, MLXCEL_FUSED_MOE_RELU2 panic on CUDA to confirm or rule out.