`8-bit quantization` support

[beratcmn]

As far as I know vllm and ray doesn't support 8-bit quantization as of now. I think it's the most viable quantization technique out there and should be implemented for faster inference and reduced memory usage.

[PenutChen]

This much-needed feature will enable a 13b model to fit into a single 24GB VRAM GPU.

[gururise]

Please consider 4-bit support as well. The new bitsandbytes library supports both 8 and 4-bit quantization.

[ehartford]

Just that 8 bit is faster than 4 bit
And vllm is about speed
It would be really nice to at least get 8 bit

[FocusLiwen]

Just that 8 bit is faster than 4 bit And vllm is about speed It would be really nice to at least get 8 bit

Why is 8 bit faster than 4 bit?

[wenmengzhou]

8 bit is a well trade-off between speed and accuracy in practice, so 8bit support is strongly required to add to vllm

[hikmet-demir]

Would love to see this happening soon.

[AntoninLeroy]

Must have feature.
Will enable using mixtral decently on a single GPU 💯

[hmellor]

Are you wanting load_in_8bit from HF or would you consider the AWQ GPTQ support sufficient?

[PenutChen]

Are you wanting load_in_8bit from HF or would you consider the AWQ GPTQ support sufficient?

@hmellor Since AWQ is becoming more popular and GPTQ is supported in vLLM, I think it's sufficient for production use. Introducing an on-the-fly quantization method, like bitsandbytes or quanto, would be more user-friendly for research purposes.

[beratcmn]

Are you wanting load_in_8bit from HF or would you consider the AWQ GPTQ support sufficient?

@hmellor cloud compute costs adds for quantizing models to AWQ and GPTQ so having an "on the go" quantization method would be incredible.

[hmellor]

I know this won't cover all situations but you could use models that have already been quantised and uploaded to Huggingface? (e.g. the almost 4000 quantised checkpoints uploaded by TheBloke https://huggingface.co/TheBloke)

[beratcmn]

@hmellor does models quantized by BnB and uploaded to hub work with vLLM?