Eval bug: Gemma4 attn_rot_k and v = 0

[vektorprime]

Name and Version

docker run -p 8080:8080 --runtime=nvidia --gpus all -v /home/user/llm/models:/models ghcr.io/ggml-org/llama.cpp:server-cuda --version

ggml_cuda_init: found 2 CUDA devices (Total VRAM: 29928 MiB):
Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes, VRAM: 24124 MiB
Device 1: NVIDIA GeForce RTX 3050, compute capability 8.6, VMM: yes, VRAM: 5804 MiB
load_backend: loaded CUDA backend from /app/libggml-cuda.so
load_backend: loaded CPU backend from /app/libggml-cpu-haswell.so
version: 8643 (f49e917)
built with GNU 14.2.0 for Linux x86_64

Operating systems

Linux (ubuntu 22)

GGML backends

CUDA

Hardware

rtx 3090 and 3050 (tried with just 3090 too)

Models

Gemma 4 31B

Problem description & steps to reproduce

I noticed the attention rotation for KV cache does not work with Gemma4, whereas it works (on this system) with Qwen3.5.

Is that expected?

llama_kv_cache_iswa: creating non-SWA KV cache, size = 120064 cells
llama_kv_cache: CUDA0 KV buffer size = 2901.94 MiB
llama_kv_cache: CUDA1 KV buffer size = 322.44 MiB
llama_kv_cache: size = 3224.38 MiB (120064 cells, 10 layers, 1/1 seqs), K (q5_0): 1612.19 MiB, V (q5_0): 1612.19 MiB
llama_kv_cache: attn_rot_k = 0
llama_kv_cache: attn_rot_v = 0
llama_kv_cache_iswa: creating SWA KV cache, size = 1536 cells
llama_kv_cache: CUDA0 KV buffer size = 1128.00 MiB
llama_kv_cache: CUDA1 KV buffer size = 72.00 MiB
llama_kv_cache: size = 1200.00 MiB ( 1536 cells, 50 layers, 1/1 seqs), K (f16): 600.00 MiB, V (f16): 600.00 MiB
llama_kv_cache: attn_rot_k = 0
llama_kv_cache: attn_rot_v = 0

EDIT PLEASE NOTE: After updating and getting both SWA and non-SWA KV caches to the SAME quant, the attn_rot_k and v are STILL showing 0.

docker run -p 8080:8080 --runtime=nvidia --gpus all -v /home/user/llm/models:/models ghcr.io/ggml-org/llama.cpp:server-cuda
-m /models/Gemma4-31B/gemma-4-31B-it-UD-Q4_K_XL.gguf
--port 8080 --host 0.0.0.0
--no-mmap --threads 8 --jinja
--cache-type-k q8_0 --cache-type-v q8_0 --flash-attn on -kvu --ctx-size 100000 -np 1
--temp 1 --top-p 0.95 --top-k 64

First Bad Commit

No response

Relevant log output

Logs

llama_kv_cache_iswa: creating non-SWA KV cache, size = 120064 cells
llama_kv_cache:      CUDA0 KV buffer size =  2901.94 MiB
llama_kv_cache:      CUDA1 KV buffer size =   322.44 MiB
llama_kv_cache: size = 3224.38 MiB (120064 cells,  10 layers,  1/1 seqs), K (q5_0): 1612.19 MiB, V (q5_0): 1612.19 MiB
**llama_kv_cache: attn_rot_k = 0
llama_kv_cache: attn_rot_v = 0**
llama_kv_cache_iswa: creating     SWA KV cache, size = 1536 cells
llama_kv_cache:      CUDA0 KV buffer size =  1128.00 MiB
llama_kv_cache:      CUDA1 KV buffer size =    72.00 MiB
llama_kv_cache: size = 1200.00 MiB (  1536 cells,  50 layers,  1/1 seqs), K (f16):  600.00 MiB, V (f16):  600.00 MiB
**llama_kv_cache: attn_rot_k = 0
llama_kv_cache: attn_rot_v = 0**

[lukasstraub2]

This is expected I think, kv cache quantization is disabled when using SWA between llama.cpp versions >= b8634 and < b8644. Upgrade to the latest version.

[vektorprime]

This is expected I think, kv cache quantization is disabled when using SWA between llama.cpp versions >= b8634 and < b8644. Upgrade to the latest version.

Oh I didn't realize the docker containers were behind in version. I will build latest now and test shortly!

[vektorprime]

I just realized I think you may have misunderstood my question. It wasn't regarding the KV quant of SWA, it was the attention rotation being disabled.

user@ubuntu:~/llm/llama.cpp/build/bin$ ./llama-cli --version
ggml_cuda_init: found 2 CUDA devices (Total VRAM: 29928 MiB):
Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes, VRAM: 24124 MiB
Device 1: NVIDIA GeForce RTX 3050, compute capability 8.6, VMM: yes, VRAM: 5804 MiB
version: 8660 (d006858)
built with GNU 11.4.0 for Linux x86_64

llama_context: CUDA_Host output buffer size = 1.00 MiB
llama_kv_cache_iswa: creating non-SWA KV cache, size = 120064 cells
llama_kv_cache: CUDA0 KV buffer size = 4484.81 MiB
llama_kv_cache: CUDA1 KV buffer size = 498.31 MiB
llama_kv_cache: size = 4983.12 MiB (120064 cells, 10 layers, 1/1 seqs), K (q8_0): 2491.56 MiB, V (q8_0): 2491.56 MiB
llama_kv_cache: attn_rot_k = 0
llama_kv_cache: attn_rot_v = 0
llama_kv_cache_iswa: creating SWA KV cache, size = 1536 cells
llama_kv_cache: CUDA0 KV buffer size = 599.25 MiB
llama_kv_cache: CUDA1 KV buffer size = 38.25 MiB
llama_kv_cache: size = 637.50 MiB ( 1536 cells, 50 layers, 1/1 seqs), K (q8_0): 318.75 MiB, V (q8_0): 318.75 MiB
llama_kv_cache: attn_rot_k = 0
llama_kv_cache: attn_rot_v = 0

[m18coppola]

The Gemma 4 SWA is not quantized and kept at full precision (see #21332). Since the KV cache is mixed precision, it's no longer able to rotate the vectors.

[vektorprime]

The Gemma 4 SWA is not quantized and kept at full precision (see #21332). Since the KV cache is mixed precision, it's no longer able to rotate the vectors.

I'm sorry, I don't fully understand this because the output I pasted earlier shows that the SWA and non-SWA KV cache are both running at Q8_0 AFTER I upgraded llama-cpp to the latest version.
llama_kv_cache_iswa: creating non-SWA KV cache, size = 120064 cells
llama_kv_cache: size = 4983.12 MiB (120064 cells, 10 layers, 1/1 seqs), K (q8_0): 2491.56 MiB, V (q8_0): 2491.56 MiB
llama_kv_cache_iswa: creating SWA KV cache, size = 1536 cells
llama_kv_cache: size = 637.50 MiB ( 1536 cells, 50 layers, 1/1 seqs), K (q8_0): 318.75 MiB, V (q8_0): 318.75 MiB

Also, that PR says it's reverted, so both cache are running at Q8_0. Am I missing something here?

[vektorprime]

Just want to clarify again, both non-SWA and SWA KV caches are using same quant (Q8_0), yet the attention rotation is still off.

llama_kv_cache: attn_rot_k = 0
llama_kv_cache: attn_rot_v = 0

[stduhpf]

I looked into it, and it looks like attn_rot is disabled because the dimension of the embeddings is not the same for every layer for Gemma4.

(ref: https://github.com/ggml-org/llama.cpp/blob/master/src/llama-kv-cache.cpp#L280 https://github.com/ggml-org/llama.cpp/blob/master/src/llama-kv-cache.cpp#L286)

Edit: It seems to still be working fine if I just comment these lines (above) out 🤔. And attn_rot is now reported as enabled.

llama_kv_cache: attn_rot_k = 1
llama_kv_cache: attn_rot_v = 1

I was expecting it to break things, but so far nothing bad happened?

[vektorprime]

I looked into it, and it looks like attn_rot is disabled because the dimension of the embeddings is not the same for every layer for Gemma4.

(ref: https://github.com/ggml-org/llama.cpp/blob/master/src/llama-kv-cache.cpp#L280 https://github.com/ggml-org/llama.cpp/blob/master/src/llama-kv-cache.cpp#L286)

Edit: It seems to still be working fine if I just comment these lines (above) out 🤔. And attn_rot is now reported as enabled.

llama_kv_cache: attn_rot_k = 1 llama_kv_cache: attn_rot_v = 1

I was expecting it to break things, but so far nothing bad happened?

Thanks for taking the time to look into this! I'm going to try and compile with those commented out and run some quick benchmarks.

[vektorprime]

/home/user/llm/llama.cpp/build/bin/llama-perplexity -m /home/user/llm/models/Gemma4-31B/gemma-4-31B-it-UD-Q4_K_XL.gguf -f /home/user/llm/wikitext-2-raw/wiki.test.raw -t 8 -c 512 -fa on --no-mmap -ngl 999 -np 1 -sm none -mg 0 --kl-divergence-base /home/user/llm/models/Gemma4-31B/gemma-4-31B-it-UD-Q4_K_XL-f16.logits.bin

Final estimate: PPL = 2331.1075 +/- 44.81276
Ran it twice just because
Final estimate: PPL = 2331.1075 +/- 44.81276

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!Testing q8_0
/home/user/llm/llama.cpp/build/bin/llama-perplexity -m /home/user/llm/models/Gemma4-31B/gemma-4-31B-it-UD-Q4_K_XL.gguf -f /home/user/llm/wikitext-2-raw/wiki.test.raw -t 8 -c 512 -fa on --cache-type-k q8_0 --cache-type-v q8_0 --no-mmap -ngl 999 -np 1 -sm none -mg 0 --kl-divergence --kl-divergence-base /home/user/llm/models/Gemma4-31B/gemma-4-31B-it-UD-Q4_K_XL-f16.logits.bin

llama_kv_cache_iswa: creating non-SWA KV cache, size = 512 cells
llama_kv_cache: CUDA0 KV buffer size = 85.00 MiB
llama_kv_cache: size = 85.00 MiB ( 512 cells, 10 layers, 4/4 seqs), K (q8_0): 42.50 MiB, V (q8_0): 42.50 MiB
llama_kv_cache: attn_rot_k = 1
llama_kv_cache: attn_rot_v = 1
llama_kv_cache_iswa: creating SWA KV cache, size = 512 cells
llama_kv_cache: CUDA0 KV buffer size = 850.00 MiB
llama_kv_cache: size = 850.00 MiB ( 512 cells, 50 layers, 4/4 seqs), K (q8_0): 425.00 MiB, V (q8_0): 425.00 MiB
llama_kv_cache: attn_rot_k = 1
llama_kv_cache: attn_rot_v = 1

====== Perplexity statistics ======
Mean PPL(Q) : 2350.798527 ± 45.195456
Mean PPL(base) : 1206.219272 ± 19.419927
Cor(ln(PPL(Q)), ln(PPL(base))): 94.51%
Mean ln(PPL(Q)/PPL(base)) : 0.667264 ± 0.006613
Mean PPL(Q)/PPL(base) : 1.948898 ± 0.012889
Mean PPL(Q)-PPL(base) : 1144.579256 ± 27.580657

====== KL divergence statistics ======
Mean KLD: 0.398962 ± 0.003085
Maximum KLD: 37.182041
99.9% KLD: 12.940373
99.0% KLD: 5.888469
95.0% KLD: 1.935292
90.0% KLD: 0.945215
Median KLD: 0.041328
10.0% KLD: 0.000152
5.0% KLD: 0.000034
1.0% KLD: 0.000002
0.1% KLD: -0.000001
Minimum KLD: -0.000068

====== Token probability statistics ======
Mean Δp: -0.020 ± 0.032 %
Maximum Δp: 99.999%
99.9% Δp: 94.778%
99.0% Δp: 46.954%
95.0% Δp: 10.145%
90.0% Δp: 2.157%
75.0% Δp: 0.019%
Median Δp: -0.000%
25.0% Δp: -0.022%
10.0% Δp: -2.270%
5.0% Δp: -10.395%
1.0% Δp: -46.770%
0.1% Δp: -94.701%
Minimum Δp: -99.997%
RMS Δp : 12.353 ± 0.085 %
Same top p: 82.717 ± 0.099 %

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Now I set LLAMA_ATTN_ROT_DISABLE=1
Next, I rerun the same exact test (KV Q8_0), and I confirm the attention rotation is off.

PENDING RESULTS SINCE MY SERVER JUST CRASHED :(

[vektorprime]

Got the results back of q8_0 without rotation. Both of the results seem bad.

llama_kv_cache_iswa: creating non-SWA KV cache, size = 512 cells
llama_kv_cache: CUDA0 KV buffer size = 85.00 MiB
llama_kv_cache: size = 85.00 MiB ( 512 cells, 10 layers, 4/4 seqs), K (q8_0): 42.50 MiB, V (q8_0): 42.50 MiB
llama_kv_cache: attention rotation force disabled (LLAMA_ATTN_ROT_DISABLE)
llama_kv_cache: attn_rot_k = 0
llama_kv_cache: attn_rot_v = 0
llama_kv_cache_iswa: creating SWA KV cache, size = 512 cells
llama_kv_cache: CUDA0 KV buffer size = 850.00 MiB
llama_kv_cache: size = 850.00 MiB ( 512 cells, 50 layers, 4/4 seqs), K (q8_0): 425.00 MiB, V (q8_0): 425.00 MiB
llama_kv_cache: attention rotation force disabled (LLAMA_ATTN_ROT_DISABLE)
llama_kv_cache: attn_rot_k = 0
llama_kv_cache: attn_rot_v = 0

====== Perplexity statistics ======
Mean PPL(Q) : 2289.579345 ± 43.936429
Mean PPL(base) : 1206.219272 ± 19.419927
Cor(ln(PPL(Q)), ln(PPL(base))): 94.36%
Mean ln(PPL(Q)/PPL(base)) : 0.640877 ± 0.006665
Mean PPL(Q)/PPL(base) : 1.898145 ± 0.012651
Mean PPL(Q)-PPL(base) : 1083.360073 ± 26.407848

====== KL divergence statistics ======
Mean KLD: 0.417162 ± 0.003120
Maximum KLD: 27.314192
99.9% KLD: 12.966664
99.0% KLD: 6.037377
95.0% KLD: 2.086692
90.0% KLD: 1.019244
Median KLD: 0.044358
10.0% KLD: 0.000165
5.0% KLD: 0.000038
1.0% KLD: 0.000003
0.1% KLD: -0.000001
Minimum KLD: -0.000074

====== Token probability statistics ======
Mean Δp: -0.001 ± 0.033 %
Maximum Δp: 99.998%
99.9% Δp: 95.188%
99.0% Δp: 48.396%
95.0% Δp: 10.583%
90.0% Δp: 2.297%
75.0% Δp: 0.022%
Median Δp: -0.000%
25.0% Δp: -0.021%
10.0% Δp: -2.241%
5.0% Δp: -10.470%
1.0% Δp: -49.296%
0.1% Δp: -95.574%
Minimum Δp: -100.000%
RMS Δp : 12.648 ± 0.086 %
Same top p: 82.197 ± 0.100 %

[vektorprime]

Both of the results seem bad. Can someone tell me if my tests were valid or not?

[stduhpf]

Hard to tell, results seems to be within margin of error. Maybe you could try with something like Q4, the effects should be more easily noticable.

[vektorprime]

Q4 seems to show good improvement on KLD with rotation:

!!!Testing Q4_0 KV cache with attention rotation off

llama_kv_cache_iswa: creating non-SWA KV cache, size = 512 cells
llama_kv_cache: CUDA0 KV buffer size = 45.00 MiB
llama_kv_cache: size = 45.00 MiB ( 512 cells, 10 layers, 4/4 seqs), K (q4_0): 22.50 MiB, V (q4_0): 22.50 MiB
llama_kv_cache: attention rotation force disabled (LLAMA_ATTN_ROT_DISABLE)
llama_kv_cache: attn_rot_k = 0
llama_kv_cache: attn_rot_v = 0
llama_kv_cache_iswa: creating SWA KV cache, size = 512 cells
llama_kv_cache: CUDA0 KV buffer size = 450.00 MiB
llama_kv_cache: size = 450.00 MiB ( 512 cells, 50 layers, 4/4 seqs), K (q4_0): 225.00 MiB, V (q4_0): 225.00 MiB
llama_kv_cache: attention rotation force disabled (LLAMA_ATTN_ROT_DISABLE)
llama_kv_cache: attn_rot_k = 0
llama_kv_cache: attn_rot_v = 0

====== Perplexity statistics ======
Mean PPL(Q) : 1573.412120 ± 29.236106
Mean PPL(base) : 1206.219272 ± 19.419927
Cor(ln(PPL(Q)), ln(PPL(base))): 90.92%
Mean ln(PPL(Q)/PPL(base)) : 0.265756 ± 0.007779
Mean PPL(Q)/PPL(base) : 1.304416 ± 0.010147
Mean PPL(Q)-PPL(base) : 367.192848 ± 14.124707

====== KL divergence statistics ======
Mean KLD: 0.947188 ± 0.004785
Maximum KLD: 39.991016
99.9% KLD: 16.055832
99.0% KLD: 8.884304
95.0% KLD: 4.420417
90.0% KLD: 2.763587
Median KLD: 0.204057
10.0% KLD: 0.000694
5.0% KLD: 0.000161
1.0% KLD: 0.000013
0.1% KLD: 0.000001
Minimum KLD: -0.000005

====== Token probability statistics ======
Mean Δp: 0.654 ± 0.048 %
Maximum Δp: 100.000%
99.9% Δp: 98.920%
99.0% Δp: 77.696%
95.0% Δp: 26.362%
90.0% Δp: 6.980%
75.0% Δp: 0.088%
Median Δp: -0.000%
25.0% Δp: -0.034%
10.0% Δp: -4.166%
5.0% Δp: -19.685%
1.0% Δp: -72.234%
0.1% Δp: -98.629%
Minimum Δp: -99.999%
RMS Δp : 18.507 ± 0.091 %
Same top p: 70.892 ± 0.119 %

!!!!!!!!!!!!!!

!!!Testing Q4_0 KV cache with attention rotation on

llama_kv_cache_iswa: creating non-SWA KV cache, size = 512 cells
llama_kv_cache: CUDA0 KV buffer size = 45.00 MiB
llama_kv_cache: size = 45.00 MiB ( 512 cells, 10 layers, 4/4 seqs), K (q4_0): 22.50 MiB, V (q4_0): 22.50 MiB
llama_kv_cache: attn_rot_k = 1
llama_kv_cache: attn_rot_v = 1
llama_kv_cache_iswa: creating SWA KV cache, size = 512 cells
llama_kv_cache: CUDA0 KV buffer size = 450.00 MiB
llama_kv_cache: size = 450.00 MiB ( 512 cells, 50 layers, 4/4 seqs), K (q4_0): 225.00 MiB, V (q4_0): 225.00 MiB
llama_kv_cache: attn_rot_k = 1
llama_kv_cache: attn_rot_v = 1

====== Perplexity statistics ======
Mean PPL(Q) : 2036.647657 ± 38.644770
Mean PPL(base) : 1206.219272 ± 19.419927
Cor(ln(PPL(Q)), ln(PPL(base))): 92.27%
Mean ln(PPL(Q)/PPL(base)) : 0.523814 ± 0.007448
Mean PPL(Q)/PPL(base) : 1.688456 ± 0.012575
Mean PPL(Q)-PPL(base) : 830.428385 ± 22.035445

====== KL divergence statistics ======
Mean KLD: 0.746116 ± 0.004258
Maximum KLD: 36.947136
99.9% KLD: 15.506844
99.0% KLD: 8.129005
95.0% KLD: 3.626889
90.0% KLD: 2.089278
Median KLD: 0.128479
10.0% KLD: 0.000438
5.0% KLD: 0.000102
1.0% KLD: 0.000009
0.1% KLD: 0.000000
Minimum KLD: -0.000109

====== Token probability statistics ======
Mean Δp: 0.165 ± 0.043 %
Maximum Δp: 99.999%
99.9% Δp: 98.075%
99.0% Δp: 69.785%
95.0% Δp: 18.829%
90.0% Δp: 4.281%
75.0% Δp: 0.043%
Median Δp: -0.000%
25.0% Δp: -0.037%
10.0% Δp: -3.754%
5.0% Δp: -17.372%
1.0% Δp: -67.516%
0.1% Δp: -98.373%
Minimum Δp: -100.000%
RMS Δp : 16.666 ± 0.090 %
Same top p: 74.781 ± 0.113 %

[Dampfinchen]

I can confirm that is the case using llama-b8672 Gemma 26b A4B Q4_K_M as well using 8 bit KV Cache.