Eval bug: Gemma 4 fails logprob comparison against itself.

[espen96]

Name and Version

ggml_cuda_init: found 2 CUDA devices (Total VRAM: 32767 MiB):
Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes, VRAM: 24575 MiB
Device 1: NVIDIA GeForce RTX 2060 SUPER, compute capability 7.5, VMM: yes, VRAM: 8191 MiB
load_backend: loaded CUDA backend from F:\llama-swap_123_windows_amd64\engines\llama.cpp verify\ggml-cuda.dll
load_backend: loaded RPC backend from F:\llama-swap_123_windows_amd64\engines\llama.cpp verify\ggml-rpc.dll
load_backend: loaded CPU backend from F:\llama-swap_123_windows_amd64\engines\llama.cpp verify\ggml-cpu-zen4.dll
version: 8681 (506200c)
built with Clang 19.1.5 for Windows x86_64

Operating systems

Windows

GGML backends

CUDA

Hardware

3090ti + AMD Ryzen 9 7900X in this setup. My second GPU, 2060Ti was not used.

Tested with both a quantized model, and BF16. Same result.

Models

gemma-4-26B-A4B, primary quants were the launch day bf16 from unsloth and the q4_k_xl from yesterday

Problem description & steps to reproduce

Run llama server with default settings, connect to the provided demo. it checks the logprobs, it hightlights missmatches.

"max_tokens": 512,
"temperature": 0,
"top_p": 1,
"repetition_penalty": 1,
"presence_penalty": 0,
"frequency_penalty": 0,
"logprobs": 10,
"stream": true,
"seed": 42

Models seem to be matching up with their own probabilities just fine usually, Gemma 4 does not seem to play ball.

I have been working on a setup to highlight various aspects of quantization, across domains etc to highlight various insights beyond KLD and PPL. However, I noticed that Gemma 4 was extremely unstable.

I got Claude to quickly throw together a setup that applies the template and checks the probabilities in an isolated system.
Qwen 3.5 9B shows no difference, and Gemma 4 26B shows a difference. I also checked Minsitral 8B, no issues.

It does not seem to be able to be even close to deterministic.

Gemma 4 26B:

Qwen 3.5 9B:

Ministral 8B:

Switching it to CPU, all is fine again.

Here is the self check setup generated for this isolated test.

selfcheck.html

First Bad Commit

No response

Relevant log output

Logs

[pwilkin]

Can you repro against the non-imatrix quant from GGML? (https://huggingface.co/ggml-org/gemma-4-26B-A4B-it-GGUF/resolve/main/gemma-4-26B-A4B-it-Q4_K_M.gguf)

[espen96]

Just did actually. it appears to pass perfectly.

it appears Barkowsi's quants also pass, downloaded after the tokenizer and template fixes, sometime the 4th.

The unsloth quants they posted after the vocab and template fixes on the 4th, broken, as well as the BF16 they posted on launch day. it also does not pass the self verification. Same with the Q5_K_XL and Q6_K_XL

So at the very least, as I don't have Unsloths quants from 7h ago, those quants appear to have something about them that breaks them, not being able to produce the same logits across two runs without 100% cpu usage.

I'm uploading the verified broken Unsloth Q4_K_XL here.

[espen96]

Been checking a few things.

Testing the GGML q4km again, it does pass, with or without n-cpu-moe. fully on gpu, fully on cpu, it passes, repeatedly.

the ggml f16 does pass on cpu. I can not test it on gpu only, but in a hybrid configuration, it does not pass.

GGML Q8_0 does pass.
Bartowski BF16, Fails in hybrid, passes on cpu.

[am17an]

Try #21566

[espen96]

Try #21566

I think that did it.

Ran 5 consecutive verification passes with the Unsloth quant that consistently showed breakage, no issue to be seen up to 1828 positions,
gglm F16, seems fine. Bartowski bf16, seems fine.