Misc. bug: [VULKAN[ [Intel] Qwen3-Coder-30B-A3B Low PP performance of Q4 and Q6 quants compared to Q8 on Intel Arc A770

[savvadesogle]

Name and Version

C:\llm\llama-cpp\VULKAN\b8149>llama-cli --version
load_backend: loaded RPC backend from C:\llm\llama-cpp\VULKAN\b8149\ggml-rpc.dll
ggml_vulkan: Found 3 Vulkan devices:
ggml_vulkan: 0 = Intel(R) Arc(TM) A770 Graphics (Intel Corporation) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: none
ggml_vulkan: 1 = Intel(R) Arc(TM) A770 Graphics (Intel Corporation) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: none
ggml_vulkan: 2 = Intel(R) Arc(TM) A770 Graphics (Intel Corporation) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: none
load_backend: loaded Vulkan backend from C:\llm\llama-cpp\VULKAN\b8149\ggml-vulkan.dll
load_backend: loaded CPU backend from C:\llm\llama-cpp\VULKAN\b8149\ggml-cpu-haswell.dll
version: 8149 (a96a112)
built with Clang 19.1.5 for Windows x86_64

Operating systems

Windows

Which llama.cpp modules do you know to be affected?

llama-bench

Command line

llama-bench -m T:\models\lmstudio-community\Qwen3.5-35B-A3B-GGUF\Qwen3.5-35B-A3B-Q6_K.gguf  -ngl 100 -fa 0,1 --mmap 1

llama-bench -m T:\models\lmstudio-community\Qwen3.5-35B-A3B-GGUF\Qwen3.5-35B-A3B-Q4_K_M.gguf  -ngl 100 -fa 0,1 --mmap 1

llama-bench -m T:\models\lmstudio-community\Qwen3.5-35B-A3B-GGUF\Qwen3.5-35B-A3B-Q8_0.gguf  -ngl 100 -fa 0,1 --mmap 1

Problem description & steps to reproduce

The PP speed for the Q8_0 model is higher than for the Q4 and Q6 quants.
600 t/s vs 200 t/s

Windows 11
Intel Arc A770 (16gb) x4
Intel xeon 2699v3 x2
Driver: 8509

Models: https://huggingface.co/lmstudio-community/Qwen3.5-35B-A3B-GGUF

First is test for 3x A770
Second: 2x A770

C:\llm\llama-cpp\VULKAN\b8149>llama-bench -m T:\models\lmstudio-community\Qwen3.5-35B-A3B-GGUF\Qwen3.5-35B-A3B-Q8_0.gguf  -ngl 100 -fa 0,1 --mmap 1
load_backend: loaded RPC backend from C:\llm\llama-cpp\VULKAN\b8149\ggml-rpc.dll
ggml_vulkan: Found 3 Vulkan devices:
ggml_vulkan: 0 = Intel(R) Arc(TM) A770 Graphics (Intel Corporation) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: none
ggml_vulkan: 1 = Intel(R) Arc(TM) A770 Graphics (Intel Corporation) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: none
ggml_vulkan: 2 = Intel(R) Arc(TM) A770 Graphics (Intel Corporation) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: none
load_backend: loaded Vulkan backend from C:\llm\llama-cpp\VULKAN\b8149\ggml-vulkan.dll
load_backend: loaded CPU backend from C:\llm\llama-cpp\VULKAN\b8149\ggml-cpu-haswell.dll
| model                          |       size |     params | backend    | ngl | fa | mmap |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -: | ---: | --------------: | -------------------: |
| qwen35moe ?B Q8_0              |  34.36 GiB |    34.66 B | Vulkan     | 100 |  0 |    1 |           pp512 |        612.14 + 1.46 |
| qwen35moe ?B Q8_0              |  34.36 GiB |    34.66 B | Vulkan     | 100 |  0 |    1 |           tg128 |         38.88 + 0.06 |
| qwen35moe ?B Q8_0              |  34.36 GiB |    34.66 B | Vulkan     | 100 |  1 |    1 |           pp512 |        614.43 + 2.65 |
| qwen35moe ?B Q8_0              |  34.36 GiB |    34.66 B | Vulkan     | 100 |  1 |    1 |           tg128 |         37.82 + 0.04 |

build: a96a1120b (8149)

C:\llm\llama-cpp\VULKAN\b8149>set GGML_VK_VISIBLE_DEVICES=2,3

C:\llm\llama-cpp\VULKAN\b8149>llama-bench -m T:\models\lmstudio-community\Qwen3.5-35B-A3B-GGUF\Qwen3.5-35B-A3B-Q6_K.gguf  -ngl 100 -fa 0,1 --mmap 1
load_backend: loaded RPC backend from C:\llm\llama-cpp\VULKAN\b8149\ggml-rpc.dll
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = Intel(R) Arc(TM) A770 Graphics (Intel Corporation) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: none
ggml_vulkan: 1 = Intel(R) Arc(TM) A770 Graphics (Intel Corporation) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: none
load_backend: loaded Vulkan backend from C:\llm\llama-cpp\VULKAN\b8149\ggml-vulkan.dll
load_backend: loaded CPU backend from C:\llm\llama-cpp\VULKAN\b8149\ggml-cpu-haswell.dll
| model                          |       size |     params | backend    | ngl | fa | mmap |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -: | ---: | --------------: | -------------------: |
| qwen35moe ?B Q6_K              |  26.55 GiB |    34.66 B | Vulkan     | 100 |  0 |    1 |           pp512 |        175.02 + 1.64 |
| qwen35moe ?B Q6_K              |  26.55 GiB |    34.66 B | Vulkan     | 100 |  0 |    1 |           tg128 |         44.18 + 0.12 |
| qwen35moe ?B Q6_K              |  26.55 GiB |    34.66 B | Vulkan     | 100 |  1 |    1 |           pp512 |        174.17 + 2.50 |
| qwen35moe ?B Q6_K              |  26.55 GiB |    34.66 B | Vulkan     | 100 |  1 |    1 |           tg128 |         42.94 + 0.06 |

build: a96a1120b (8149)

C:\llm\llama-cpp\VULKAN\b8149>llama-bench -m T:\models\lmstudio-community\Qwen3.5-35B-A3B-GGUF\Qwen3.5-35B-A3B-Q4_K_M.gguf  -ngl 100 -fa 0,1 --mmap 1
load_backend: loaded RPC backend from C:\llm\llama-cpp\VULKAN\b8149\ggml-rpc.dll
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = Intel(R) Arc(TM) A770 Graphics (Intel Corporation) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: none
ggml_vulkan: 1 = Intel(R) Arc(TM) A770 Graphics (Intel Corporation) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: none
load_backend: loaded Vulkan backend from C:\llm\llama-cpp\VULKAN\b8149\ggml-vulkan.dll
load_backend: loaded CPU backend from C:\llm\llama-cpp\VULKAN\b8149\ggml-cpu-haswell.dll
| model                          |       size |     params | backend    | ngl | fa | mmap |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -: | ---: | --------------: | -------------------: |
| qwen35moe ?B Q4_K - Medium     |  19.71 GiB |    34.66 B | Vulkan     | 100 |  0 |    1 |           pp512 |        242.54 + 1.90 |
| qwen35moe ?B Q4_K - Medium     |  19.71 GiB |    34.66 B | Vulkan     | 100 |  0 |    1 |           tg128 |         49.18 + 0.07 |
| qwen35moe ?B Q4_K - Medium     |  19.71 GiB |    34.66 B | Vulkan     | 100 |  1 |    1 |           pp512 |        240.96 + 4.06 |
| qwen35moe ?B Q4_K - Medium     |  19.71 GiB |    34.66 B | Vulkan     | 100 |  1 |    1 |           tg128 |         47.45 + 0.03 |

build: a96a1120b (8149)

[TabNahida]

Vulkan PP is way slower than SYCL.

[HumerousGorgon]

Vulkan PP is way slower than SYCL.

How is TG on SYCL though?

[engrtipusultan]

I do not think what you observe is a bug. Q8 is simplest math hence prompt processing is the highest on Q8. But token generation is slow.

Here is analysis on my machine for vulkan. Based on your back-end and GPU it might be different for you little bit.

Conclusion:

Overall Rank	Quant Type	PP TOPS (Rank)	TG TOPS (Rank)	Average Rank
1	iq2_xs	1.52 (1)	0.22 (1)	1.0
2	iq2_xxs	1.52 (1)	0.18 (2)	1.5
3	iq4_nl	1.50 (3)	0.13 (6)	4.5
4	iq2_s	1.43 (7)	0.18 (2)	4.5
5	mxfp4	1.47 (5)	0.13 (6)	5.5
6	iq3_xxs	1.43 (7)	0.17 (4)	5.5
7	q5_0	1.44 (6)	0.12 (8)	7.0
8	q4_0	1.31 (13)	0.15 (5)	9.0
9	iq3_s	1.42 (9)	0.12 (8)	8.5
10	q8_0	1.50 (3)	0.08 (13)	8.0
11	q4_K	1.40 (10)	0.11 (10)	10.0
12	q5_K	1.37 (11)	0.11 (10)	10.5
13	iq4_xs	1.32 (12)	0.11 (10)	11.0
14	q6_K	1.27 (14)	0.08 (13)	13.5

Based on:

bash  ./quant-bench -d vulkan0
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon Graphics (RADV RENOIR) (radv) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 0 | matrix cores: none
Using device: Vulkan0

=== Prompt Processing (Prefill) Phase (Batch Size = 512) ===

Quant	Time (us)	TOPS
q4_0	45852.47	1.31
q4_K	43057.02	1.40
q5_0	41751.17	1.44
q5_K	44029.87	1.37
q6_K	47377.60	1.27
q8_0	40007.54	1.50
iq2_xxs	39476.48	1.52
iq2_xs	39648.50	1.52
iq2_s	42163.56	1.43
iq3_xxs	42061.97	1.43
iq3_s	42290.59	1.42
iq4_nl	40171.84	1.50
iq4_xs	45535.99	1.32
mxfp4	40768.00	1.47

=== Token Generation (Decoding) Phase (Batch Size = 1) ===

Quant	Time (us)	TOPS	Eff. BW (GB/s)
q4_0	796.95	0.15	41.54
q4_K	1029.23	0.11	32.16
q5_0	961.21	0.12	42.08
q5_K	1103.46	0.11	36.65
q6_K	1433.84	0.08	33.65
q8_0	1513.28	0.08	41.28
iq2_xxs	641.39	0.18	23.72
iq2_xs	522.21	0.22	32.64
iq2_s	637.26	0.18	29.63
iq3_xxs	683.46	0.17	33.00
iq3_s	959.73	0.12	26.37
iq4_nl	913.31	0.13	36.25
iq4_xs	1053.45	0.11	29.68
mxfp4	875.12	0.13	35.73

[savvadesogle]

I do not think what you observe is a bug. Q8 is simplest math hence prompt processing is the highest on Q8. But token generation is slow.

Hello. There is a problem. There shouldn't be a 3x difference.
Here is a test example of another model.

C:\llm\llama-cpp\VULKAN\b8149>llama-bench -m T:\models\unsloth\gpt-oss-20b-GGUF\gpt-oss-20b-Q4_K_M.gguf  -ngl 100 -fa 0,1 --mmap 1
load_backend: loaded RPC backend from C:\llm\llama-cpp\VULKAN\b8149\ggml-rpc.dll
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = Intel(R) Arc(TM) A770 Graphics (Intel Corporation) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: none
ggml_vulkan: 1 = Intel(R) Arc(TM) A770 Graphics (Intel Corporation) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: none
load_backend: loaded Vulkan backend from C:\llm\llama-cpp\VULKAN\b8149\ggml-vulkan.dll
load_backend: loaded CPU backend from C:\llm\llama-cpp\VULKAN\b8149\ggml-cpu-haswell.dll
| model                          |       size |     params | backend    | ngl | fa | mmap |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -: | ---: | --------------: | -------------------: |
| gpt-oss 20B Q4_K - Medium      |  10.81 GiB |    20.91 B | Vulkan     | 100 |  0 |    1 |           pp512 |      1101.15 + 12.44 |
| gpt-oss 20B Q4_K - Medium      |  10.81 GiB |    20.91 B | Vulkan     | 100 |  0 |    1 |           tg128 |         63.17 + 0.14 |
| gpt-oss 20B Q4_K - Medium      |  10.81 GiB |    20.91 B | Vulkan     | 100 |  1 |    1 |           pp512 |      1140.49 + 10.22 |
| gpt-oss 20B Q4_K - Medium      |  10.81 GiB |    20.91 B | Vulkan     | 100 |  1 |    1 |           tg128 |         62.98 + 0.16 |

build: a96a1120b (8149)

C:\llm\llama-cpp\VULKAN\b8149>llama-bench -m T:\models\unsloth\gpt-oss-20b-GGUF\gpt-oss-20b-Q6_K.gguf  -ngl 100 -fa 0,1 --mmap 1
load_backend: loaded RPC backend from C:\llm\llama-cpp\VULKAN\b8149\ggml-rpc.dll
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = Intel(R) Arc(TM) A770 Graphics (Intel Corporation) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: none
ggml_vulkan: 1 = Intel(R) Arc(TM) A770 Graphics (Intel Corporation) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: none
load_backend: loaded Vulkan backend from C:\llm\llama-cpp\VULKAN\b8149\ggml-vulkan.dll
load_backend: loaded CPU backend from C:\llm\llama-cpp\VULKAN\b8149\ggml-cpu-haswell.dll
| model                          |       size |     params | backend    | ngl | fa | mmap |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -: | ---: | --------------: | -------------------: |
| gpt-oss 20B Q6_K               |  11.20 GiB |    20.91 B | Vulkan     | 100 |  0 |    1 |           pp512 |        991.13 + 6.00 |
| gpt-oss 20B Q6_K               |  11.20 GiB |    20.91 B | Vulkan     | 100 |  0 |    1 |           tg128 |         60.99 + 0.08 |
| gpt-oss 20B Q6_K               |  11.20 GiB |    20.91 B | Vulkan     | 100 |  1 |    1 |           pp512 |       1011.46 + 5.40 |
| gpt-oss 20B Q6_K               |  11.20 GiB |    20.91 B | Vulkan     | 100 |  1 |    1 |           tg128 |         61.39 + 0.11 |

build: a96a1120b (8149)

C:\llm\llama-cpp\VULKAN\b8149>llama-bench -m T:\models\unsloth\gpt-oss-20b-GGUF\gpt-oss-20b-Q8_0.gguf  -ngl 100 -fa 0,1 --mmap 1
load_backend: loaded RPC backend from C:\llm\llama-cpp\VULKAN\b8149\ggml-rpc.dll
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = Intel(R) Arc(TM) A770 Graphics (Intel Corporation) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: none
ggml_vulkan: 1 = Intel(R) Arc(TM) A770 Graphics (Intel Corporation) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: none
load_backend: loaded Vulkan backend from C:\llm\llama-cpp\VULKAN\b8149\ggml-vulkan.dll
load_backend: loaded CPU backend from C:\llm\llama-cpp\VULKAN\b8149\ggml-cpu-haswell.dll
| model                          |       size |     params | backend    | ngl | fa | mmap |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -: | ---: | --------------: | -------------------: |
| gpt-oss 20B Q8_0               |  11.27 GiB |    20.91 B | Vulkan     | 100 |  0 |    1 |           pp512 |        941.58 + 9.38 |
| gpt-oss 20B Q8_0               |  11.27 GiB |    20.91 B | Vulkan     | 100 |  0 |    1 |           tg128 |         60.85 + 0.42 |
| gpt-oss 20B Q8_0               |  11.27 GiB |    20.91 B | Vulkan     | 100 |  1 |    1 |           pp512 |        963.99 + 3.13 |
| gpt-oss 20B Q8_0               |  11.27 GiB |    20.91 B | Vulkan     | 100 |  1 |    1 |           tg128 |         61.14 + 0.27 |

build: a96a1120b (8149)

[savvadesogle]

Vulkan PP is way slower than SYCL.

realy?)

SYCL

Vulkan

[engrtipusultan]

I do not think what you observe is a bug. Q8 is simplest math hence prompt processing is the highest on Q8. But token generation is slow.

Hello. There is a problem. There shouldn't be a 3x difference. Here is a test example of another model.

Maybe you are correct. Experts can way in on this more. But this has been my conclusion in past. That is why I use Q8 normally. I am just sharing my experience. On longer conversation you need good pp and Q8 provides best accuracy and best PP on my machine.

One point though you have given example of GPT OSS which is natively trained on MXFP4. When quantized to Q5 and Q6 am not sure how it suppose to behave.
Just to be sure if you have good internet and space you can try any other model that is not trained on 4 bit.

[0cc4m]

The variance on quant performance on Intel is not really expected, no. I don't have a good explanation, Intel GPUs still perform much less consistently than AMD and Nvidia. It took me a long time to get any kind of usable performance out of them. While I can for example optimize for AMD and see improvements on Nvidia, Intel often needs special attention or it can even regress. It's been a lot of work. I think the drivers are not as mature (especially the Linux one) and the hardware is still relatively fresh and not as well understood in comparison.

[savvadesogle]

@0cc4m
And that's a shame. Because Intel's performance could have been excellent. But Intel still doesn't seem to be paying enough attention to llama.cpp.🤷🥺