8bit support

[mymusise]

Hi, will vllm support 8bit quantization? Like https://github.com/TimDettmers/bitsandbytes

In HF, we can run a 13B LLM on a 24G GPU with load_in_8bit=True.

Although PageAttention can save 25% of GPU memory, but we have to deploy a 13B LLM on a 26G GPU, at least.

In the cloud, v100-32G is more expensive than A5000-24G 😭

Is there any way to save video memory usage? 😭

[mymusise]

same with #214

[gururise]

Would love to see bitsandbytes integration to load models in 8 and 4-bit quantized mode.

[generalsvr]

Quantization support is crucial. 8 and 4 bit support is a must

[cabbagetalk]

why I use fastchat-vllm to inference vicuna-13B，It took 75 G of video memory(A800, 80G) @mymusise

[mymusise]

why I use fastchat-vllm to inference vicuna-13B，It took 75 G of video memory(A800, 80G) @mymusise

@cabbagetalk you can add gpu_memory_utilization=0.4 to free your memory

[proceduralia]

This would especially useful for running the new meta-llama/Llama-2-70b-chat-hf models!

[boxter007]

Hi, will vllm support 8bit quantization? Like https://github.com/TimDettmers/bitsandbytes

In HF, we can run a 13B LLM on a 24G GPU with load_in_8bit=True.

Although PageAttention can save 25% of GPU memory, but we have to deploy a 13B LLM on a 26G GPU, at least.

In the cloud, v100-32G is more expensive than A5000-24G 😭

Is there any way to save video memory usage? sob

在这里看到熟人啊

[yhyu13]

Does vllm support it now?

[louis-csm]

Any fix for this issue?

[warvyvr]

Hi guys, do you have plan to support it?

[qashzar]

any fix for integrating bitsandbytes?

[hmellor]

No fix, but the feature request is #4033