Repository navigation
Misc. bug: Severe drop in prompt processing speed after b10780 on Vulkan, RDNA3 #28752
Description
Name and Version
❯ llama-server --version
version: 0.3.0-dev (build 10781, commit unknown)
built with GNU 16.2.1 for Linux x86_64
Operating systems
Linux
Which llama.cpp modules do you know to be affected?
llama-server
Command line
llama-server --device Vulkan0 -b 2048 -ub 1024 --port 10000 --slot-save-path /home/llamacpp/slots/ -cram 32768 -bs --models-max 1 -np 1 --sleep-idle-seconds 3600 --models-preset /home/llamacpp/llms/config.ini
[Qwen 3.8 27B Q4_K_M]
c = 60000
n-gpu-layers = 99
spec-draft-ngl = 99
spec-type = draft-mtp
spec-draft-n-max = 3
model-draft = /home/llamacpp/llms/qwen27b/mtp-Qwen3.8-27B-Q4_0.gguf
model = /home/llamacpp/llms/qwen27b/Qwen3.8-27B-UD-Q4_K_M.gguf
ctk = q8_0
ctv = q8_0
ctkd = q8_0
ctvd = q8_0
chat-template-file = /home/llamacpp/llms/qwen27b/chat_template.jinjaProblem description & steps to reproduce
Versions b10781 and later has greatly reduced prompt processing speeds at depth compared to the previous release. Reverting to release b10780 restores performance.
First Bad Commit
Appears to be caused by commit c7bda03, #28190
Relevant log output
Logs
b10798: prompt processing, n_tokens = 49985, progress = 1.00, t = 153.87 s / 324.85 tokens per second
b10781: prompt processing, n_tokens = 49985, progress = 1.00, t = 141.61 s / 352.97 tokens per second
prompt processing, n_tokens = 49985, progress = 1.00, t = 132.14 s / 378.27 tokens per second
b10780: prompt processing, n_tokens = 49985, progress = 1.00, t = 75.82 s / 659.26 tokens per second
prompt processing, n_tokens = 49985, progress = 1.00, t = 75.33 s / 663.57 tokens per second
b10754: prompt processing, n_tokens = 49985, progress = 1.00, t = 75.86 s / 658.91 tokens per second
Activity
0cc4m commentedon Sep 11, 2026
@Nathanw1014
Nathanw1014 commentedon Sep 11, 2026
@ComputerGuy4157 what GPU do you have.
Can you please run another test at different prompt size, or with MTP off.
#28190 engages a scratch buffer that puts additional memory pressure on the GPU (around ~200MB at that prompt length, on the mentioned model, ~400MB with MTP on)
ComputerGuy4157 commentedon Sep 11, 2026
@Nathanw1014
GPU is Radeon 7900XTX on Mesa 26.2.2. During testing there is ~6GB VRAM free, so I do not believe vram overcommit is an issue.
ComputerGuy4157 commentedon Sep 12, 2026
I've been fiddling around with this, and it seems the commit also makes the desktop very laggy during prompt processing at depth. Any graphical effects that have to be processed by the GPU (such as just having the llama webui open) cause prompt processing performance to plummet. However, b10780 seems to not be affected.
I've captured this in some GPU usage graphs, showing what GPU usage looks like at the end of prompt processing (and start of tg)
b10781, ui open
prompt eval time = 123707.34 ms / 49989 tokens ( 2.47 ms per token, 404.09 tokens per second)
eval time = 75309.66 ms / 1695 tokens ( 44.46 ms per token, 22.49 tokens per second)
b10781, ui not rendered
prompt eval time = 80765.46 ms / 49989 tokens ( 1.62 ms per token, 618.94 tokens per second)
eval time = 73522.96 ms / 1655 tokens ( 44.45 ms per token, 22.50 tokens per second)
b10780, ui open
prompt eval time = 73800.06 ms / 49989 tokens ( 1.48 ms per token, 677.36 tokens per second)
eval time = 84714.65 ms / 1892 tokens ( 44.80 ms per token, 22.32 tokens per second)
b10780, ui not rendered
prompt eval time = 73877.86 ms / 49989 tokens ( 1.48 ms per token, 676.64 tokens per second)
eval time = 89543.08 ms / 1998 tokens ( 44.84 ms per token, 22.30 tokens per second)
mpapili commentedon Sep 12, 2026
+1'ing this. I can reproduce on an Rx 7900xtx
The following is from mid-august (apologies, I did not save the commit):
Command
Before (Mid-August Build)
Today (acecd56 ; after b10780)
40 t/s decode down to 6 t/s and a non-insignificant hit to prompt-processing as well.
kwilliams90 commentedon Sep 13, 2026
Reproduced on a different RDNA3 card (Radeon PRO W7800, gfx1100), Windows 11 Pro 26200, Adrenalin driver 32.0.21041.1000 (2026-01-07), official Windows Vulkan release zips, no rebuild.
Model: Qwen3.8-27B Q6_K GGUF (27.3B params,
qwen35arch), MTP draft enabled (--spec-type draft-mtp --spec-draft-n-max 2). Same flags on both builds:Measurement: one cold
/v1/chat/completionsrequest with a 63,432-token prompt (cache_promptoff), readingtimings.prompt_ms/timings.prompt_per_secondfrom the response.b10809 is 1.55x slower on prompt processing for the same request; token generation and VRAM are unchanged. On b10809,
-ub 1024(425 s) and-ub 2048(432 s) did not help, and switching the KV cache to f16/f16 only brought it to 354 s (179 tok/s), so neither batch size nor KV quantization explains the gap.One caveat: the 31-token warm-up prompt took 845 ms on b10780 versus 265 ms on b10809 in a single sample, so very short prompts may have gone the other way; I did not repeat that case.
DanielGarciaTheCoder commentedon Sep 14, 2026
Adding an RDNA4 data point with a bisect that shows two steps, not one. Windows, AMD proprietary driver.
Environment
AMD Radeon RX 9070 XT (AMD proprietary driver) | uma: 0 | fp16: 1 | bf16: 1 | fp4: 0 | warp size: 64 | shared memory: 32768 | int dot: 1 | matrix cores: KHR_coopmatllama-bNNNNN-bin-win-vulkan-x64.ziprelease assets, no local buildCommand
Bisect over release zips, pp4096 t/s with q8_0 KV
So on this coopmat1 device the first step (#25494) already costs about 20 percent at depth 0, and #28190 turns it into 35 percent at depth 0. At depth 8192 (pp4096 after 8192 tokens of context) the current build reads 592 t/s against 1509 t/s on b9637, a 61 percent loss. Decode is unchanged at every depth (tg128 122 vs 121 at depth 8192). Three launches per build agree within 1 percent.
KV type on b10956, same model
b9637 with q4_1 reads 3036 / 1645, so with q4_1 the current build is 2 to 4 percent faster than June. q4_1 is the workaround I am using.
Env switches on b10956 with q8_0 KV: GGML_VK_DISABLE_FUSION, GGML_VK_DISABLE_GRAPH_OPTIMIZE, GGML_VK_DISABLE_COOPMAT, GGML_VK_DISABLE_HOST_VISIBLE_VIDMEM, GGML_VK_DISABLE_MMVQ, GGML_VK_DISABLE_ASYNC, GGML_VK_SERIALIZE_SUBMISSIONS all read 1922 to 1931 t/s. None of them changes it.
One more observation that may or may not belong here: f16 KV is also slow on b10956 on this device (1896 at depth 0), even though the f16 scratch path is not merged as far as I can tell. And on this driver a 512-token prefill batch runs about 6 times slower than 1024 on every build back to b9637 (pp512 548, pp1024 3644, pp2048 3380, pp4096 2978 on b9637), which looks like a driver shader choice for that one shape and is independent of these PRs.
Happy to run more cells or a patched build on request.
0cc4m commentedon Sep 16, 2026
I cannot reproduce this on AMD R9700 or on AMD Radeon 8060S on Linux. I don't have any RDNA3 dGPU to test.
salatets commentedon Sep 20, 2026
Not able to reproduce on RX 7900XTX, mesa 26.2.3, CachyOS
llama-bench -m Qwen3.8-27B-UD-Q4_K_XL.gguf -ngl 999 -fa 0,1 -p 2048,4096 --device Vulkan0
b10516
b10780
b11063
Nathanw1014 commentedon Sep 21, 2026
@salatets can you please try with Q8_0,
-ctk q8_0 -ctv q8_0salatets commentedon Sep 22, 2026
@Nathanw1014
Here test with
-ctk q8_0 -ctv q8_0, but without-fa 0, becausellama_bench: error: failed to create context with model 'Qwen3.8-27B-UD-Q4_K_XL.gguf'10516
b10780
b11063
ComputerGuy4157 commentedon Sep 23, 2026
@salatets Could you please test at longer depths e.g.
-d 0-65536+16384salatets commentedon Sep 23, 2026
@ComputerGuy4157
No regression, still
q8
b10516
10780
11063
full
b10516
b10780
b11063
tczank commentedon Sep 24, 2026
Exact single-line A/B on RX 7900 XTX + Mesa 26.2.3: no reproduction —
c7bda03is neutral-to-faster hereBuilt
c7bda03and its parent0df974d7from source (identical config,-DGGML_VULKAN=ON, native) and ran them back-to-back on one rig.Environment: RX 7900 XTX 24 GB, Mesa 26.2.3-arch1.1 (RADV), Arch Linux; model
Qwen3.8-27B-GSQ-RCO-IQ3_XXS(9.72 GiB),-ngl 999 --device Vulkan0.llama-bench,
-r 3, pp t/s:0df974d7c7bda03-fa 0-fa 0-fa 1-fa 1-fa 1 -ctk q8_0 -ctv q8_0-fa 1 -ctk q8_0 -ctv q8_0Depth (llama-server, reporter's regime):
-c 65536 -ctk q8_0 -ctv q8_0 -ngl 99 -np 1, MTP off, 64-token probes with unique prompt markers; pp t/s, two reps per build:c7bda03At the reporter's decisive depth: parent mean 623 vs
c7bda03mean 665 (+6.7 % forc7bda03), consistent across both reps — versus the reported 663 → 353/378 t/s. The ±4–5 % swings on the default-f16-KV configs (where the quantized-dequant gate isn't the operative factor) size the run-order/warm-up noise here; note the 4 K server pair flips sign between reps under the same ordering, so a "second-run advantage" doesn't explain the consistent 50 K delta.Reading: with q8_0 KV the context only creates when the FA path is engaged (as @salatets found with
-fa 0 -ctk q8_0→ context-creation failure), so the depth runs do traverseggml_vk_flash_attn— on 26.2.3 the widened engagement (path active mid-cache instead of only when full) is neutral-to-beneficial rather than a regression. Combined with @salatets' non-repro (same GPU, Mesa 26.2.3), the split lines up with the driver stack: reporter = Mesa 26.2.2, Windows reports = Adrenalin — the scratch/staging buffer this path engages (~200–400 MB, per @Nathanw1014) seems to behave pathologically on those drivers.Suggestion: re-test on Mesa 26.2.3 — if the cliff disappears, this is a driver-version interaction rather than something to fix in llama.cpp (worst case: an opt-out to keep the old gate condition as a workaround).
Caveats: desktop-compositor load during pp not replicated (the OP's lag observation); my model/quant differs from the OP's (IQ3_XXS vs Q4_K_M) — each A/B pair is identical; MTP off (the OP's mtp-off rows regressed too).
ComputerGuy4157 commentedon Oct 14, 2026
After a while of poking at different settings, I noticed that ReBAR was disabled. After enabling it, the regression in speed disappears.
Two runs, rebar on and off, same prompt of ~60000 tokens on latest (b11606)
The last two entries for rebar off are missing because the driver crashes with vk::Queue::submit: ErrorDeviceLost
Marking as resolved.