CUDA: fix slow q4_0/q5_0 V dequantization on Blackwell with CUDA 12.8

[ilyas-mallah]

Overview

The vector FA kernel with q4_0 or q5_0 V cache gets a 128 to 336 byte stack frame on CUDA 12.8 for Blackwell (sm_120a, sm_100a, sm_101a). Older architectures and CUDA 12.9 and 13.0 are not affected.

This PR changes dequantize_V_q4_0 and dequantize_V_q5_0 so they read the dequantized word with shifts, instead of through an int8_t pointer to it, which nvcc 12.8 kept in local memory. The result is that the stack frame for those kernels is removed.

On an RTX 5090 built with CUDA 12.8.1 (the version the cuda Docker images use), the FA op with a q4_0 cache becomes 2.1x as fast and decode gets 1.7x faster at 16k context and 2.1x at 32k. q8_0 is unchanged. Built with CUDA 13.0, master already runs at this speed.

This is only done on Blackwell (__CUDA_ARCH__ >= 1000); other architectures, HIP and MUSA keep the pointer read.

Additional information

Performance (RTX 5090, CUDA 12.8.1, llama-bench -fa 1 -ctk/-ctv q8_0,q4_0 -p 0 -n 128 -d 0,4096,16384,32768)

Table generated with scripts/compare-llama-bench.py. V type equals K type.

GPU	Model	K type	Test	t/s master	t/s branch	Speedup
RTX 5090	qwen3 8B Q4_K_M	q4_0	tg128	215.63	235.00	1.09
RTX 5090	qwen3 8B Q4_K_M	q4_0	tg128@d4096	164.41	212.66	1.29
RTX 5090	qwen3 8B Q4_K_M	q4_0	tg128@d16384	100.54	171.85	1.71
RTX 5090	qwen3 8B Q4_K_M	q4_0	tg128@d32768	65.07	134.39	2.07
RTX 5090	qwen3 8B Q4_K_M	q8_0	tg128	234.31	234.35	1.00
RTX 5090	qwen3 8B Q4_K_M	q8_0	tg128@d4096	211.85	211.93	1.00
RTX 5090	qwen3 8B Q4_K_M	q8_0	tg128@d16384	169.06	168.94	1.00
RTX 5090	qwen3 8B Q4_K_M	q8_0	tg128@d32768	131.24	131.09	1.00

FA op alone (head size 128, 8 K/V heads, GQA 4, q4_0): 88.1 to 41.2 us at 16k K/V rows, 164.8 to 77.1 us at 32k.

Stack frame per nvcc version (-Xptxas -v, D=128, one column)

nvcc	sm_86 q4_0	sm_89 q4_0	sm_90a q4_0	sm_120a q4_0	sm_120a q8_0
12.8.93	0 B	0 B	0 B	128 B	0 B
12.9.86				0 B	0 B
13.0.88	0 B	0 B		0 B	0 B

With this PR, q4_0 and q5_0 on sm_120a with nvcc 12.8 drop to the frames all types have (the two-column and D=64 softcap kernels). The stack is already in the PTX from nvcc 12.8, so it comes from the front end, not ptxas.

test-backend-ops -o FLASH_ATTN_EXT -p type_K=q passes on the RTX 5090 (1728/1728).

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES. Claude Code was used to investigate the slowdown and what change would be required to drop the stack frame. I ended up choosing this fix, reviewed the change and the results, and wrote this description myself.

[JohannesGaessler]

Performance

Table generated with scripts/compare-llama-bench.py

GPU	Model	K type	V type	Microbatch size	Test	t/s b11458	t/s b4e5089	Speedup
RTX 3090	llama 8B Q4_0	q4_0	q4_0	1	pp1024@d32768	54.80	55.02	1.00
RTX 3090	llama 8B Q4_0	q4_0	q4_0	2	pp1024@d32768	90.23	89.98	1.00
RTX 3090	llama 8B Q4_0	q4_0	q4_0	4	pp1024@d32768	160.62	158.42	0.99
RTX 3090	llama 8B Q4_0	q5_0	q5_0	1	pp1024@d32768	49.97	43.20	0.86
RTX 3090	llama 8B Q4_0	q5_0	q5_0	2	pp1024@d32768	87.26	79.60	0.91
RTX 3090	llama 8B Q4_0	q5_0	q5_0	4	pp1024@d32768	153.77	136.67	0.89
RTX 4090	llama 8B Q4_0	q4_0	q4_0	1	pp1024@d32768	112.75	113.23	1.00
RTX 4090	llama 8B Q4_0	q4_0	q4_0	2	pp1024@d32768	188.10	189.25	1.01
RTX 4090	llama 8B Q4_0	q4_0	q4_0	4	pp1024@d32768	214.92	215.04	1.00
RTX 4090	llama 8B Q4_0	q5_0	q5_0	1	pp1024@d32768	97.10	97.54	1.00
RTX 4090	llama 8B Q4_0	q5_0	q5_0	2	pp1024@d32768	166.32	165.64	1.00
RTX 4090	llama 8B Q4_0	q5_0	q5_0	4	pp1024@d32768	232.31	232.32	1.00
RTX 5090	llama 8B Q4_0	q4_0	q4_0	1	pp1024@d32768	165.62	165.61	1.00
RTX 5090	llama 8B Q4_0	q4_0	q4_0	2	pp1024@d32768	273.54	280.10	1.02
RTX 5090	llama 8B Q4_0	q4_0	q4_0	4	pp1024@d32768	361.24	361.57	1.00
RTX 5090	llama 8B Q4_0	q5_0	q5_0	1	pp1024@d32768	145.28	146.68	1.01
RTX 5090	llama 8B Q4_0	q5_0	q5_0	2	pp1024@d32768	246.83	248.95	1.01
RTX 5090	llama 8B Q4_0	q5_0	q5_0	4	pp1024@d32768	381.25	381.17	1.00

I'm not seeing the reported speedup with

> $ nvcc --version                                                                                                                                            [±b4e5089ea ●(✹)]
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2026 NVIDIA Corporation
Built on Tue_Jun_09_02:43:40_PM_PDT_2026
Cuda compilation tools, release 13.3, V13.3.73
Build cuda_13.3.r13.3/compiler.38244171_0

[ilyas-mallah]

Thanks for testing! The slowdown only happens with CUDA 12.8, which is what the default cuda Docker images use, so no difference with 13.3 is expected. 12.8 compiles that byte-pointer read for Blackwell through local memory, newer versions don't.

You're right about the q5_0 regression on the 3090. I pushed a change so the shifts are only used on Blackwell and every other architecture keeps the original code. I checked that the generated code for the older architectures is now identical to master.

[JohannesGaessler]

@CISC can we just update the docker image to CUDA 12.9? I would like to avoid having to work around compiler bugs like this in the backend code.

[CISC]

@CISC can we just update the docker image to CUDA 12.9? I would like to avoid having to work around compiler bugs like this in the backend code.

IIRC there were some complaints about doing that as some cloud platforms were locked to 12.8.

Edit: Ah yes, #21438 .. maybe this has changed by now?

[ilyas-mallah]

@CISC can we just update the docker image to CUDA 12.9? I would like to avoid having to work around compiler bugs like this in the backend code.

The Ubuntu CUDA 12.8 release builds have the same problem on Blackwell, so they'd need the bump too. Anyone building from source with 12.8 would also still hit it.