cuda 12

[patrafter1999]

Hi Guys,

Thanks heaps for your work. I'm really impressed by the performance.

I'm just curious if you guys have CUDA 12 support on your roadmap? I have just grabbed an A100 instance with CUDA 12.2 installed. I was so excited and realised vLLM supports up to CUDA 11.8.

Thanks!

[WoosukKwon]

Hi @patrafter1999, thanks for your interest! Actually, it's because there's no PyTorch wheels pre-built with CUDA 12. vLLM will automatically support CUDA 12 once PyTorch 2.1.0 is released (with CUDA 12 support).

For now, you can install vLLM inside an NVIDIA PyTorch docker.

[vince62s]

I don't understand this answer. Pytorch nightly is already cuda 12.1 with 2.1.0dev with a wheel.
but it still triggers an error when installing vllm

Never mind, it works with python setup.py develop

[tanguofu]

I don't understand this answer. Pytorch nightly is already cuda 12.1 with 2.1.0dev with a wheel. but it still triggers an error when installing vllm

Never mind, it works with python setup.py develop

is there some plan to fix this? thanks.

[ferrybaltimore]

2.1 stable is already available

[LRY89757]

2.1 stable is already available

really? But I can compile the vllm with the cuda 12.1, pytorch 2.1, but I can' t run it correctly, because of it:

[sxk000]

CUDA Version: 12.2 , it is not work

python setup.py install
Traceback (most recent call last):
File "/apply/app/vllm/setup.py", line 63, in
raise RuntimeError(
RuntimeError: Cannot find CUDA_HOME. CUDA must be available to build the package.

python setup.py develop
Traceback (most recent call last):
File "/apply/app/vllm/setup.py", line 63, in
raise RuntimeError(
RuntimeError: Cannot find CUDA_HOME. CUDA must be available to build the package.

how to install ? thanks

[simon-mo]

CUDA 12 was supported in recent version.