Difference of gguf conversion ?

[razvanab]

Are there any differences in the GGUF conversion between stable-diffusion.cpp and the rest?

Almost any GGUF I tried doesn't work in stable-diffusion.cpp and needs the main model to be converted with the sd.exe specifically to work.

[stduhpf]

As far as I know only Stable Diffusion 3/3.5 GGUFs are incompatible between ComfyUI GGUF and stable-diffusion.cpp, because ComfyUI-GGUF slightly modifies some tensor's shape for an obscure reason).

The main difference for all other models is that the GGUFs made with sd.exe convert all tensors to the same quant type, except a few that are kept at full precision, while GGUFs made with other tools tend to use a mixture of different quant types within a same file. One of the reason some GGUFs might not work well is that the version of GGML used by stable-diffusion.cpp is a bit out of date, and some types are poorly supported by most backends (Example: BF16 on Vulkan).

[razvanab]

I see.

I tried with Chroma and T5, and I got some errors.
chroma-unlocked-v40-Q8_0.gguf

Both
flan_t5_xxl_TE-only_Q6_K.gguf

and
t5-v1_1-xxl-encoder-Q6_K.gguf

log.txt

[stduhpf]

Ah yes, I forgot about T5. The T5xxl quants you're trying to use are made for llama.cpp. Llama.cpp renames all tensors to their own convention, while stable-diffusion.cpp uses the original naming convention of the model.

It should work with https://huggingface.co/Green-Sky/flux.1-schnell-GGUF/blob/main/t5xxl_q4_k.gguf or https://huggingface.co/Green-Sky/flux.1-schnell-GGUF/blob/main/t5xxl_q8_0.gguf, for example, or there's also this patch: #654 I made earlier that would help support the one you tried.

[razvanab]

Nice.
Thank you.

[Green-Sky]

@razvanab #654 has been merged.