GGUF support

[viktor-ferenczi]

Motivation

AWQ is nice, but if you want more control over the bit depth (thus VRAM usage), then GGUF may be a better option. A wide range of models are available from TheBloke at various bit depths, so everybody can use the biggest one which can fit into their GPUs.

I cannot find a high-throughput batch inference engine which can load GGUF, maybe there is none. (vLLM cannot load it either.)

Related resources

https://github.com/ggerganov/llama.cpp

https://huggingface.co/TheBloke

[casper-hansen]

FYI, high throughput is hard when using quantized models in general, regardless of which framework. But if you can manage to run with batch size (data parallelism) of less than 8 or ideally less than 4, then it will increase throughput. At 16, there is no gain with quantized models because of how quantized inference works.

[viktor-ferenczi]

Does the AWQ implementation support higher than 4 bits per weight, for example 8 bits?

[casper-hansen]

Does the AWQ implementation support higher than 4 bits per weight, for example 8 bits?

Not yet. It’s 4-bit only at the moment.

[viktor-ferenczi]

Thank you.

Could you please point me to some technical details what makes it hard to implement high throughput (batching, caching) and using quantization (unpacking quantized data on demand) at the same time? These seem to be pretty orthogonal for me.

I'm learning into the LLM (transformer) implementations and have past coding experience. Therefore I'm really interested in knowing the details.

[casper-hansen]

Thank you.

Could you please point me to some technical details what makes it hard to implement high throughput (batching, caching) and using quantization (unpacking quantized data on demand) at the same time? These seem to be pretty orthogonal for me.

I'm learning into the LLM (transformer) implementations and have past coding experience. Therefore I'm really interested in knowing the details.

Yes. The way quantization works is that weights are quantized to 4 bits. Then at inference time, you run dequantization to FP16 to be able to perform matrix multiplication. This dequantization is the essence of any quantized model and is why quantized models generally struggle with large batch sizes because it becomes compute-bound doing dequantization rather than the actual matrix multiplication. At batch size 1, you are memory bound, which speeds up your inference by a great deal.

[viktor-ferenczi]

On your documentation page there is an excellent high level summary on how to add support for a new model.

Could you please write down a few bullet points on where to look in the code (high level) if I want to add a new input format (GGUF)?

I would start by searching for all the code involved in reading model configuration and parameters.

I'm also aware that for 8 bit operation at least the dequantization code will need to be extended as well.

Also, what do you think about AWQ? Should we expect 8 bit support be added to AWQ in the near future? Because that would make any work on GGUF pointless.

If I would work on GGUF support it would certainly be based on the AWQ branch. Which is the correct branch to look at? (There seem to be multiple and I'm a bit confused.)

Thank you!

[bet0x]

This would be a good starting point: https://github.com/oobabooga/text-generation-webui/blob/9331ab4798f392ee0634a4194a6bae370afd435f/modules/metadata_gguf.py

oobabooga/text-generation-webui@9331ab4#diff-4f7989a17db10e843396d051093357685cd8b4f69e186734e5be93e293689ae8

[viktor-ferenczi]

8-bit GGUF could be a perfect input format for the upcoming W8A8 inference mode, see #1112

[sh1ng]

Thank you.
Could you please point me to some technical details what makes it hard to implement high throughput (batching, caching) and using quantization (unpacking quantized data on demand) at the same time? These seem to be pretty orthogonal for me.
I'm learning into the LLM (transformer) implementations and have past coding experience. Therefore I'm really interested in knowing the details.

Yes. The way quantization works is that weights are quantized to 4 bits. Then at inference time, you run dequantization to FP16 to be able to perform matrix multiplication. This dequantization is the essence of any quantized model and is why quantized models generally struggle with large batch sizes because it becomes compute-bound doing dequantization rather than the actual matrix multiplication. At batch size 1, you are memory bound, which speeds up your inference by a great deal.

Could you please elaborate more on this or point to some code?
At first glance, the amount of work is proportional to a batch size in both cases(dequantization, matrix-multiplication). Or at larger batches, matrix multiplication is implemented more efficiently (with some CUDA tricks) while dequantization stays the same?

[casper-hansen]

The amount of work scales linearly. The problem is when you increase batch size too much because then your GPU will be 100% utilized just doing matrix multiplication. Once that happens, the dequantization overhead will start showing in how fast you can run inference compared to FP16.

This is also why finding algorithms that can quantize to W4A4 (very hard) or W8A8 (hard) is essential for higher throughput because you remove the need for dequantization since you can run natively on tensor cores.

[sh1ng]

I understand the motivation of W4A4 and W8A8 as everything can be done solely in INT4/INT8.
But what I don't fully understand is the following

load weights in fp16  -> matrix multiplication in fp16

load weights in int4 or int8 -> dequantization to fp16 -> matrix multiplication in fp16

if all the above blocks scale linearly(performance-wise) with batch size the second scenario must always be faster for any batch size if it's faster for batch_size=1. So I expect some non-linearity in one of the steps.

[casper-hansen]

This is the difference between memory-bound and compute-bound. At small batch sizes, you are memory bound meaning that you are limited by how fast you can pass the model’s weights around. This makes quantized models faster.

However, when we talk about large batch sizes, we move away from being memory bound. It is no longer an issue of how fast we can transport the weights but rather how much time we spend doing computations. You have to think of it as being stuck computing matmuls rather than waiting for weights to be transported through memory.

[sh1ng]

@casper-hansen thanks for the clarification. I'm still trying to connect the dots.

Is loading->dequantization->multiplication fuzed into a single kernel? Could you point me to some source code?

[casper-hansen]

Weight loading happens at startup time and then it’s transported through registers. This process is not really transparent but it all happens in the quantization kernel that you can find in csrc.