Question: K/V Quantization q5_0 slower than q4_0 and q8_0

[winstonma]

Git commit

I am running version b8625

Operating systems

Linux

GGML backends

Vulkan

Problem description & steps to reproduce

I am running Qwen3.5-0.8B-Q4_K_M.gguf (with flash-att on). Trying the benchmark the result of the new KV cache quantization.

K/V Quantization	Token Generation per second
f16	91
q8_0	88
q4_0	88
q5_0	75

Should I expect the performance of q5_0 sit between q4_0 and q8_0?

Modify the ctk and ctv to different value

llama-cli -m ~/model/Qwen3.5-0.8B-Q4_K_M.gguf -ctk q8_0 -ctv q8_0 --reasoning off --flash-attn on

First Bad Commit

No response

Compile command

I am using CachyOS default compile. The build flag is in PKGBUILD.

Relevant log output

llama_context: Vulkan_Host  output buffer size =     0.95 MiB
llama_kv_cache: layer   0: filtered
llama_kv_cache: layer   1: filtered
llama_kv_cache: layer   2: filtered
llama_kv_cache: layer   3: dev = Vulkan0
llama_kv_cache: layer   4: filtered
llama_kv_cache: layer   5: filtered
llama_kv_cache: layer   6: filtered
llama_kv_cache: layer   7: dev = Vulkan0
llama_kv_cache: layer   8: filtered
llama_kv_cache: layer   9: filtered
llama_kv_cache: layer  10: filtered
llama_kv_cache: layer  11: dev = Vulkan0
llama_kv_cache: layer  12: filtered
llama_kv_cache: layer  13: filtered
llama_kv_cache: layer  14: filtered
llama_kv_cache: layer  15: dev = Vulkan0
llama_kv_cache: layer  16: filtered
llama_kv_cache: layer  17: filtered
llama_kv_cache: layer  18: filtered
llama_kv_cache: layer  19: dev = Vulkan0
llama_kv_cache: layer  20: filtered
llama_kv_cache: layer  21: filtered
llama_kv_cache: layer  22: filtered
llama_kv_cache: layer  23: dev = Vulkan0
llama_kv_cache:    Vulkan0 KV buffer size =     0.00 MiB
llama_kv_cache: size = 1056.00 MiB (262144 cells,   6 layers,  1/1 seqs), K (q5_0):  528.00 MiB, V (q5_0):  528.00 MiB
llama_kv_cache: attn_rot_k = 1
llama_kv_cache: attn_rot_v = 1
llama_memory_recurrent, layer   0: dev = Vulkan0
llama_memory_recurrent, layer   1: dev = Vulkan0
llama_memory_recurrent, layer   2: dev = Vulkan0
llama_memory_recurrent: layer   3: skipped
llama_memory_recurrent, layer   4: dev = Vulkan0
llama_memory_recurrent, layer   5: dev = Vulkan0
llama_memory_recurrent, layer   6: dev = Vulkan0
llama_memory_recurrent: layer   7: skipped
llama_memory_recurrent, layer   8: dev = Vulkan0
llama_memory_recurrent, layer   9: dev = Vulkan0
llama_memory_recurrent, layer  10: dev = Vulkan0
llama_memory_recurrent: layer  11: skipped
llama_memory_recurrent, layer  12: dev = Vulkan0
llama_memory_recurrent, layer  13: dev = Vulkan0
llama_memory_recurrent, layer  14: dev = Vulkan0
llama_memory_recurrent: layer  15: skipped
llama_memory_recurrent, layer  16: dev = Vulkan0
llama_memory_recurrent, layer  17: dev = Vulkan0
llama_memory_recurrent, layer  18: dev = Vulkan0
llama_memory_recurrent: layer  19: skipped
llama_memory_recurrent, layer  20: dev = Vulkan0
llama_memory_recurrent, layer  21: dev = Vulkan0
llama_memory_recurrent, layer  22: dev = Vulkan0
llama_memory_recurrent: layer  23: skipped
llama_memory_recurrent:    Vulkan0 RS buffer size =    19.27 MiB
llama_memory_recurrent: size =   19.27 MiB (     1 cells,  24 layers,  1 seqs), R (f32):    1.27 MiB, S (f32):   18.00 MiB
llama_context: enumerating backends
llama_context: backend_ptrs.size() = 2

[0cc4m]

Not implemented, see #21029

[0cc4m]

Actually, which GPU were you using? It is implemented for Nvidia with NV_coopmat2.

[pjsgsy]

I can say on that model on my side, moving from Q4/8 to the others (I tested again just a few days ago with latest build), dramatically tanks performance for me, but, I think it was expected in my case, as my GPU does not likely have the same support in CUDA for those, RTX 2050 (ampere). I put it down to the extra software quant steps needed.

[0cc4m]

This is about Vulkan, not CUDA.

[winstonma]

Actually, which GPU were you using? It is implemented for Nvidia with NV_coopmat2.

I am using AMD 880M iGPU. Llama.cpp vulkan.

Not implemented, see #21029

Thanks for the info

[sorenibrahim96-coder]

Yeah, in theory q5_0 should sit between q4_0 and q8_0, but on Vulkan/AMD that’s not guaranteed. Some paths aren’t well optimized, so q5_0 can actually be slower.

…

On Thu, Apr 2, 2026, 8:50 AM Winston Ma ***@***.***> wrote: *winstonma* left a comment (ggml-org/llama.cpp#21295) <#21295 (comment)> Actually, which GPU were you using? It is implemented for Nvidia with NV_coopmat2. I am using AMD 880M iGPU. Llama.cpp vulkan. Not implemented, see #21029 <#21029> Thanks for the info — Reply to this email directly, view it on GitHub <#21295?email_source=notifications&email_token=CAOPJAEUDS766UZYTRUSDVD4TZV3NA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTIMJXHAYDGOJQGY3KM4TFMFZW63VKON2WE43DOJUWEZLEUVSXMZLOOSWGM33PORSXEX3DNRUWG2Y#issuecomment-4178039066>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/CAOPJAF77Z6YO4LTCEN273T4TZV3NAVCNFSM6AAAAACXKK2GNGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DCNZYGAZTSMBWGY> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[0cc4m]

Should be resolved.