GPU consumption

[David-Lee-1990]

when i load 13b llama in HF, GPU usage is about 26G.

However, when load 13b llama in vllm, GPU usage is about 73G.

Is this ususal?

[trannhatquy]

I load the opt 125M in the vllm api and it takes 21GB on RTX 6000, it's strange.

[rahuldshetty]

I load the opt 125M in the vllm api and it takes 21GB on RTX 6000, it's strange.

Observe the same issue with A100 20Gi profile. Using the opt-125 eats up complete GPU memory.
Seems like a case of memory leak ?

[irasin]

vllm will allocate 90% GPU memory for model inference and kv_cache blocks. So in A100 case, it will use at least 0.9 * 81920 = 73728 MiB

[rahuldshetty]

vllm will allocate 90% GPU memory for model inference and kv_cache blocks. So in A100 case, it will use at least 0.9 * 81920 = 73728 MiB

Any way to limit the GPU memory usage ? (if we can tradeoff between throughput/memory)

[irasin]

vllm will allocate 90% GPU memory for model inference and kv_cache blocks. So in A100 case, it will use at least 0.9 * 81920 = 73728 MiB

Any way to limit the GPU memory usage ? (if we can tradeoff between throughput/memory)

You can restrict the gpu usage with the gpu_memory_utilization, https://github.com/vllm-project/vllm/blob/main/vllm/engine/arg_utils.py#L27

[David-Lee-1990]

vllm will allocate 90% GPU memory for model inference and kv_cache blocks. So in A100 case, it will use at least 0.9 * 81920 = 73728 MiB

Any way to limit the GPU memory usage ? (if we can tradeoff between throughput/memory)

You can restrict the gpu usage with the gpu_memory_utilization, https://github.com/vllm-project/vllm/blob/main/vllm/engine/arg_utils.py#L27

Thanks so much. But what if the needed gpu exceed the predefined gpu_memory_utilization? will it raise OOM or automatically use more gpu memory?

[irasin]

Since vllm support continuous batching, it will automatically schedule the requests to run in each iteration, so oom will not happen.
However, if you do something like parallel sampling, you may see oom happen if there is no enough cpu_cache blocks.

[zhuohan123]

Please refer to #241 for memory usage! We will add this to our document.