Is it normal that ROCm+HIPBLAS produces different results than on CPU or breaks completely?

Hello. I did some perplexity tests while investigating issue with Llama-3. Initially the issue was that Llama-3 base 70b model outputs garbage with small quants with iMatrix. Can't find any information regarding that ROCm possibly causes corruption.

GPU test (RX 7600 + RX 7600 XT)

https://huggingface.co/mradermacher/Meta-Llama-3-70B-i1-GGUF/tree/main
Meta-Llama-3-70B.i1-Q2_K.gguf prints [1]-nan,[2]-nan,[3]-nan,[4]-nan with -ngl 30 or 0 (prints garbage unless -ngl 0)
https://huggingface.co/mradermacher/Meta-Llama-3-70B-GGUF/tree/main
Meta-Llama-3-70B.Q2_K.gguf - seems OK, [1]4.1839,[2]4.7300,[3]4.2751,[4]4.6444,[5]4.6942,[6]5.0426,[7]5.1405,[8]5.4747
Final estimate: PPL = 5.9315 +/- 0.03553

Pure CPU test

Meta-Llama-3-70B.i1-Q2_K.gguf with pure CPU 'perplexity' build (146 seconds per 512 tokens - ETA 26 hours 55.67 minutes)
[1]6.3962,[2]7.1886,[3]6.9886,[4]7.3853,[5]7.8924,[6]8.2982,[7]8.8956,[8]9.3799, (can't wait for many hours, stopped)
Meta-Llama-3-70B.Q2_K.gguf (static Q2_K):
[1]4.1675,[2]4.6952,[3]4.2374,[4]4.6452,[5]4.6677,[6]5.0459,[7]5.1258,[8]5.4649,^C
It's slightly better than on ROCm but the difference is very small.

I also found strange holes in the imatrix.dat that was used:

But the author seems uninterested in discussing that.

[slaren]

I also get nan ppl with that model with CUDA, so it does not seem specific to ROCm.

@slaren does the model itself works, in a sense, that it doesn't output garbage? Because that was mine issue with AMD initially.

[slaren]

If it looks ok with CPU, then it is probably an issue in the CUDA/HIP backend. I don't know why the nans are being generated, we would need to investigate.

[JohannesGaessler]

I think this is not an issue with the code but rather with the model from that particular repository. The repository says it took the importance matrix from another repository which in turn says nothing about how the importance matrix was calculated. Using a few hours of my 6x RTX 4090 machine I calculated an importance matrix on 2.6 million tokens of Wikitext 103. The q2_K quant created using that particular importance matrix works fine.

If it looks ok with CPU, then it is probably an issue in the CUDA/HIP backend. I don't know why the nans are being generated, we would need to investigate.

It's because CUDA/HIP hip use IEEE 754 half precision floats which have a reduced numerical range to the single precision floats used on the CPU backend. I very much expect that if the weights were converted to BF16 instead of FP16 then you at least wouldn't get NaN. Alternatively, with LLAMA_CUDA_FORCE_MMQ you may also get non-NaN results since the accumulator is FP32. In any case, look at the perplexity of the quants with/without the importance matrix; clearly there is something wrong with the importance matrix quants even on CPU.

[slaren]

I already tried disabling FP16 mat mul and LLAMA_CUDA_FORCE_MMQ and it still results in nans. The model is likely corrupted, I think there are nans in the block scales, but I don't know why it doesn't affect the CPU backend.

[JohannesGaessler]

Anyways, I've uploaded the importance matrix I generated: https://huggingface.co/JohannesGaessler/llama.cpp_importance_matrices

Hmm, could this be related?
#6804
I'm not sure where these special tokens are coming from though

[slaren]

So this has nothing to do with FP16 precision. The block scales of blk.21.ffn_up.weight have nans, which results in nans when performing the matrix multiplication. In the CPU backend, the result is then quantized again to q8_k for the next matrix multiplitcation, which dutifully eats the nans and turns them into zeros. And so it looks like it works somewhat, but it really doesn't.

I don't know how the nans got there in the first place, but the model is not valid.

[JohannesGaessler]

Good to know, thanks for investigating.

Thank you, I managed to generate IQ2_XS and even IQ2_XXS with the imatrix provided by JohannesGaessler. Not sure how bad the perplexity is:
Q2_K without imat: Final estimate: PPL = 5.9315 +/- 0.03553
IQ2_XS: Final estimate: PPL = 6.3082 +/- 0.03853
IQ2_XXS: Final estimate: PPL = 7.2195 +/- 0.04552

@slaren you mean that imatrix has NaNs, or the original Llama-3 model?

[slaren]

I was talking specifically about the Meta-Llama-3-70B.i1-Q2_K.gguf model that produces nans with CUDA and HIP.

Ah sorry, I misread your message.

Btw. Wouldn't it make sense to check for NaNs and other bad values in the model/imatrix when they are loaded/generated? I think it'd only take a fraction of second for Q quants but it could be beneficial to prevent spreading broken models?

[slaren]

Yes, I think that would be good.