ARM aarch-64 server build failed (host OS: Ubuntu22.04.3)

[zhudy]

do as: https://docs.vllm.ai/en/latest/getting_started/installation.html

1. docker run --gpus all -it --rm --ipc=host nvcr.io/nvidia/pytorch:23.10-py3
2. git clone https://github.com/vllm-project/vllm.git
3. cd vllm
4. pip install -e .

here is the details in side the docker instance:
root@f8c2e06fbf8b:/mnt/vllm# pip install -e .
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Obtaining file:///mnt/vllm
Installing build dependencies ... done
Checking if build backend supports build_editable ... done
Getting requirements to build editable ... error
error: subprocess-exited-with-error

× Getting requirements to build editable did not run successfully.
│ exit code: 1
╰─> [22 lines of output]
/tmp/pip-build-env-4xoxai9j/overlay/local/lib/python3.10/dist-packages/torch/nn/modules/transformer.py:20: UserWarning: Failed to initialize NumPy: No module named 'numpy' (Triggered internally at /pytorch/torch/csrc/utils/tensor_numpy.cpp:84.)
device: torch.device = torch.device(torch._C._get_default_device()), # torch.device('cpu'),
No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda'
:142: UserWarning: Unsupported CUDA/ROCM architectures ({'6.1', '7.2', '8.7', '5.2', '6.0'}) are excluded from the TORCH_CUDA_ARCH_LIST env variable (5.2 6.0 6.1 7.0 7.2 7.5 8.0 8.6 8.7 9.0+PTX). Supported CUDA/ROCM architectures are: {'7.5', '8.0', '9.0', '7.0', '8.6+PTX', '9.0+PTX', '8.6', '8.0+PTX', '8.9+PTX', '8.9', '7.0+PTX', '7.5+PTX'}.
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in
main()
File "/usr/local/lib/python3.10/dist-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
File "/usr/local/lib/python3.10/dist-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 132, in get_requires_for_build_editable
return hook(config_settings)
File "/tmp/pip-build-env-4xoxai9j/overlay/local/lib/python3.10/dist-packages/setuptools/build_meta.py", line 441, in get_requires_for_build_editable
return self.get_requires_for_build_wheel(config_settings)
File "/tmp/pip-build-env-4xoxai9j/overlay/local/lib/python3.10/dist-packages/setuptools/build_meta.py", line 325, in get_requires_for_build_wheel
return self._get_build_requires(config_settings, requirements=['wheel'])
File "/tmp/pip-build-env-4xoxai9j/overlay/local/lib/python3.10/dist-packages/setuptools/build_meta.py", line 295, in _get_build_requires
self.run_setup()
File "/tmp/pip-build-env-4xoxai9j/overlay/local/lib/python3.10/dist-packages/setuptools/build_meta.py", line 311, in run_setup
exec(code, locals())
File "", line 297, in
File "", line 267, in get_vllm_version
NameError: name 'nvcc_cuda_version' is not defined. Did you mean: 'cuda_version'?
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build editable did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

[notice] A new release of pip is available: 23.2.1 -> 23.3.1
[notice] To update, run: python -m pip install --upgrade pip

[zhudy]

Actually, the nvcc is ok to run as these:

root@f8c2e06fbf8b:/mnt/vllm# nvcc -v
nvcc fatal : No input files specified; use option --help for more information
root@f8c2e06fbf8b:/mnt/vllm# nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Tue_Aug_15_22:10:07_PDT_2023
Cuda compilation tools, release 12.2, V12.2.140
Build cuda_12.2.r12.2/compiler.33191640_0

[zhudy]

there is cuda:

root@f8c2e06fbf8b:/mnt/vllm# echo $CUDA_HOME
/usr/local/cuda

root@f8c2e06fbf8b:/mnt/vllm# type nvcc
nvcc is /usr/local/cuda/bin/nvcc

github.com/vllm# python3 -c "import torch; print(torch.cuda.is_available()); print(torch.version);"
True
2.1.0a0+32f93b1

[yexing]

add

nvcc_cuda_version = get_nvcc_cuda_version(CUDA_HOME) 

to setup.py at line 268

[cyc00518]

@yexing @zhudy
Excuse me. I face the same problem.
I cloned vllm into my project.
and add
nvcc_cuda_version = get_nvcc_cuda_version(CUDA_HOME)
to setup.py at line 268

But still have same problem. Did I mislead something?

[Wetzr]

I have the same problem and would be glad if there would be any help.
Setup:
Aarch64 GH200
OS: Ubuntu 22.04.3 LTS (Jammy Jellyfish)
nvcc: nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Wed_Nov_22_11:03:34_PST_2023
Cuda compilation tools, release 12.3, V12.3.107
Build cuda_12.3.r12.3/compiler.33567101_0
cuda home: /usr/local/cuda
Torch: 2.2.0a0+81ea7a4

I am running inside the nvidia pytorch_23.12 Container.

[haileyschoelkopf]

Got it working with the changes in this branch: https://github.com/haileyschoelkopf/vllm/tree/aarm64-dockerfile , with built dockerfiles here: https://hub.docker.com/r/haileysch/vllm-aarch64-base https://hub.docker.com/r/haileysch/vllm-aarch64-openai hopefully this'll be helpful to others!

[tuanhe]

do as: https://docs.vllm.ai/en/latest/getting_started/installation.html

1. docker run --gpus all -it --rm --ipc=host nvcr.io/nvidia/pytorch:23.10-py3
2. git clone https://github.com/vllm-project/vllm.git
3. cd vllm
4. pip install -e .

here is the details in side the docker instance: root@f8c2e06fbf8b:/mnt/vllm# pip install -e . Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com Obtaining file:///mnt/vllm Installing build dependencies ... done Checking if build backend supports build_editable ... done Getting requirements to build editable ... error error: subprocess-exited-with-error

× Getting requirements to build editable did not run successfully. │ exit code: 1 ╰─> [22 lines of output] /tmp/pip-build-env-4xoxai9j/overlay/local/lib/python3.10/dist-packages/torch/nn/modules/transformer.py:20: UserWarning: Failed to initialize NumPy: No module named 'numpy' (Triggered internally at /pytorch/torch/csrc/utils/tensor_numpy.cpp:84.) device: torch.device = torch.device(torch._C._get_default_device()), # torch.device('cpu'), No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda' :142: UserWarning: Unsupported CUDA/ROCM architectures ({'6.1', '7.2', '8.7', '5.2', '6.0'}) are excluded from the TORCH_CUDA_ARCH_LIST env variable (5.2 6.0 6.1 7.0 7.2 7.5 8.0 8.6 8.7 9.0+PTX). Supported CUDA/ROCM architectures are: {'7.5', '8.0', '9.0', '7.0', '8.6+PTX', '9.0+PTX', '8.6', '8.0+PTX', '8.9+PTX', '8.9', '7.0+PTX', '7.5+PTX'}. Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in main() File "/usr/local/lib/python3.10/dist-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main json_out['return_val'] = hook(**hook_input['kwargs']) File "/usr/local/lib/python3.10/dist-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 132, in get_requires_for_build_editable return hook(config_settings) File "/tmp/pip-build-env-4xoxai9j/overlay/local/lib/python3.10/dist-packages/setuptools/build_meta.py", line 441, in get_requires_for_build_editable return self.get_requires_for_build_wheel(config_settings) File "/tmp/pip-build-env-4xoxai9j/overlay/local/lib/python3.10/dist-packages/setuptools/build_meta.py", line 325, in get_requires_for_build_wheel return self._get_build_requires(config_settings, requirements=['wheel']) File "/tmp/pip-build-env-4xoxai9j/overlay/local/lib/python3.10/dist-packages/setuptools/build_meta.py", line 295, in _get_build_requires self.run_setup() File "/tmp/pip-build-env-4xoxai9j/overlay/local/lib/python3.10/dist-packages/setuptools/build_meta.py", line 311, in run_setup exec(code, locals()) File "", line 297, in File "", line 267, in get_vllm_version NameError: name 'nvcc_cuda_version' is not defined. Did you mean: 'cuda_version'? [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip. error: subprocess-exited-with-error

× Getting requirements to build editable did not run successfully. │ exit code: 1 ╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

[notice] A new release of pip is available: 23.2.1 -> 23.3.1 [notice] To update, run: python -m pip install --upgrade pip

HI, guys , had you solved the issue ?

[cyc00518]

@tuanhe
Still face same problem,
Anyone know vllm support aarch-64 now?

[drikster80]

Had a similar problem on the GH200 (aarch64 Grace CPU).
Similar to @haileyschoelkopf, I updated the Dockerfile and requirements to work with v0.5.1. Here is the forked version:
https://github.com/drikster80/vllm/tree/gh200-docker

Main issues that needed to be overcome:

• Use Nvidia's Pytorch container due to PyTorch not supporting ARM64. (specifically nvcr.io/nvidia/pytorch:24.04-py3 to ensure PyTorch 2.3 build and latest optimizations (e.g. Lightning-Thunder). Release Notes for 24.04-py3
• xformers hangs on pip install. Not sure why (maybe just taking forever to compile?)
• Triton needs to be installed from source
• vllm-flash-attn needs to be built from source
• Comment out "torch", "xformers", and "vllm-flash-attn" in requirements files (handling that in the Dockerfile directly).

For future updating, you can see the changes here: drikster80@359fd4f

[ZihaoZhou]

Thank you all.

I have built the image using the script provided by @drikster80, and it takes about 12 hours (most of the time is spent for mamba builder and xformers). So to save time for others, I have made the image public at https://hub.docker.com/r/zihaokevinzhou/vllm-aarch64-openai . I have validated it works well for my personal hosting of fp8 quantized version of llama-3-70b.

[drikster80]

@ZihaoZhou, thank you.

It normally only takes ~80 min on my system. 12 hrs seems excessive. I'm working on an update for v0.5.2, but haven't gotten the new flash-infer to build yet. I'll update the script when that's solved and post back here.

I haven't been uploading since the container is ~33GB. It looks like the one you uploaded is 13GB? Is that just from native compression? I'm sure there are some ways to cut it down (e.g. remove some of the build artifacts from the last image?).

[cyc00518]

@ZihaoZhou
You should have appeared earlier!

@drikster80
In fact, I am also using GH200, and today I used yours forked version to build it.

The step that took me the most time was:
RUN python3 setup.py bdist_wheel --dist-dir=dist which took a total of 40 minutes.
Installing Triton also took a very long time.

Additionally, for the xformers part, I spent an entire afternoon, and it also seemed to be stuck there.
So in the end, I commented out this part.

Now, vllm is successfully running on GH200, thanks to your selfless contribution!

May I ask, regarding the Docker image on aarch64, compared to the original version, is the main difference just commenting out the items you mentioned in the requirements.txt?
Why is this necessary?

[drikster80]

@cyc00518 You can see the list of full changes here: main...drikster80:vllm:gh200-docker

Effectively, xformers and vllm-flash-attention don't release ARM64 wheels, so those need to be built from source. Also, since Nvidia's PyTorch container already containers torch, torchvision, and some other stuff, those need to be commented out in the requirements file. The only 3 files that are changes are Dockerfile, requirements-build.txt, and requirements-cuda.txt.

As a side note, if you're using a GH200 bare metal, you might also want to checkout my auto-install for GH200s. Getting it setup with optimizations, NCCL, OFED, for high-speed distributed training/inference was a pain, so automated it for people to use or reference: https://github.com/drikster80/gh200-Ubuntu-22.04-autoinstall

Main issues that needed to be overcome:

• Use Nvidia's Pytorch container due to PyTorch not supporting ARM64. (specifically nvcr.io/nvidia/pytorch:24.04-py3 to ensure PyTorch 2.3 build and latest optimizations (e.g. Lightning-Thunder). Release Notes for 24.04-py3
• xformers hangs on pip install. Not sure why (maybe just taking forever to compile?)
• Triton needs to be installed from source
• vllm-flash-attn needs to be built from source
• Comment out "torch", "xformers", and "vllm-flash-attn" in requirements files (handling that in the Dockerfile directly).

[cyc00518]

@drikster80
Thank you very much for your patient replies.

I have learned a lot, and I also appreciate the additional information you provided！

[drikster80]

Updated the aarch64 remote branch to v0.5.2: https://github.com/drikster80/vllm/tree/gh200-docker

Pushed up a GH200 specific version (built for SM 9.0+PTX) to https://hub.docker.com/r/drikster80/vllm-gh200-openai

Building a more generic version now and will update this comment when complete.