Installing with ROCM

[baderex]

Hello,

I'm trying to install VLLM on AMD server. However unable to build the package because CUDA is not installed. Is their anyway we can configure it to work with ROCM instead?

!pip install vllm

Error:
RuntimeError: Cannot find CUDA_HOME. CUDA must be available in order to build the package.

ROCM is installed and verified

PyTorch 2-0-ROCm

[zhuohan123]

AMD GPUs are not supported now and we just added this into our roadmap.

CC @naed90

[ccbadd]

My server is running a pair of MI100's and I would love to try this out. Please add rocm support.

[naed90]

My server is running a pair of MI100's and I would love to try this out. Please add rocm support.

Hey! Currently planning to work on this. If you can reach out, we’d appreciate it (got some questions): Dean.leitersdorf@gmail.com

[jamestwhedbee]

Hey @naed90, I would also love to try this out on MI100s. I am going to go ahead and shoot you an email as well.

[chymian]

is it possible to get clblast/VULKAN support, for a bigger range of supported cards, as well?
there are a lot of older cards, which are only partly/not supported by the latest ROCm, drivers.

[GdRottoli]

The enhancement for AMD Support was added two weeks ago. Do you have any info about the current state of this issue? Thanks!

[ehartford]

Very interested in this feature

[smiraldr]

@GdRottoli can we have some documentation on this feature too? Don't see any docs on how to use vllm with MI100s

[ehartford]

I also want to use vllm on mi100s

[smiraldr]

If anyone can point me towards the code which achieve this - I can experiment and I'd also be willing to contribute to the docs !

[ardfork]

The major blocker for a ROCm HIP port is xFormers (and flash attention). Without that, it shouldn't be that hard to hipify vLLM.

Edit: After taking a closer look, vLLM also use a lot of inline PTX assembly, that will also be annoying to port.