[vLLM] The ray serve using vLLM example on the website does not work.

[power-puff-gg]

What happened + What you expected to happen

On following all the steps mentioned in the tutorial: https://docs.ray.io/en/latest/serve/tutorials/vllm-example.html, I get the following error:

`(ServeReplica:default:VLLMDeployment pid=1328933)     return any(model.name == model_name for model in self.base_model_paths)
(ServeReplica:default:VLLMDeployment pid=1328933)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(ServeReplica:default:VLLMDeployment pid=1328933)   File "/root/miniconda3/envs/venv/lib/python3.11/site-packages/vllm/entrypoints/openai/serving_engine.py", line 665, in <genexpr>

(ServeReplica:default:VLLMDeployment pid=1328933)     return any(model.name == model_name for model in self.base_model_paths)
(ServeReplica:default:VLLMDeployment pid=1328933)                ^^^^^^^^^^
(ServeReplica:default:VLLMDeployment pid=1328933) AttributeError: 'str' object has no attribute 'name'`

The model I was trying out was the one mentioned in the tutorial: NousResearch/Meta-Llama-3-8B-Instruct

Versions / Dependencies

vllm==0.6.6.post
ray==2.41.0

Reproduction script

Create llm.py and query.py as mentioned in the tutorial and run the following commands:

serve run llm:build_app model="NousResearch/Meta-Llama-3-8B-Instruct" tensor-parallel-size=2 accelerator="GPU"

python query.py

Issue Severity

High: It blocks me from completing my task.

[pcmoritz]

Thanks for reporting, this has just been fixed in #50192, can you take the updated code from there?

[power-puff-gg]

Hello @pcmoritz ,

Thank you for addressing the problem.
I tried to install it using pip by specifying the commit ID:

pip install git+https://github.com/ray-project/ray.git@d173688d7bfde61f0f571fb234a4d2d1acfa1b55

The commit points to #50155 which was pushed after #50192

It did not work and I am am getting the following error:

Error Log:

Collecting git+https://github.com/ray-project/ray.git@d173688d7bfde61f0f571fb234a4d2d1acfa1b55
  Cloning https://github.com/ray-project/ray.git (to revision d173688d7bfde61f0f571fb234a4d2d1acfa1b55) to /tmp/pip-req-build-bhdaft1e
  Running command git clone --filter=blob:none --quiet https://github.com/ray-project/ray.git /tmp/pip-req-build-bhdaft1e
  Running command git rev-parse -q --verify 'sha^d173688d7bfde61f0f571fb234a4d2d1acfa1b55'
  Running command git fetch -q https://github.com/ray-project/ray.git d173688d7bfde61f0f571fb234a4d2d1acfa1b55
  Running command git checkout -q d173688d7bfde61f0f571fb234a4d2d1acfa1b55
  Resolved https://github.com/ray-project/ray.git to commit d173688d7bfde61f0f571fb234a4d2d1acfa1b55
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error
  
  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [35 lines of output]
      Traceback (most recent call last):
        File "/root/miniconda3/envs/py311vLLMLatest/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/root/miniconda3/envs/py311vLLMLatest/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/root/miniconda3/envs/py311vLLMLatest/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
          return hook(config_settings)
                 ^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-sib7ewi4/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 334, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=[])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-sib7ewi4/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 304, in _get_build_requires
          self.run_setup()
        File "/tmp/pip-build-env-sib7ewi4/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 522, in run_setup
          super().run_setup(setup_script=setup_script)
        File "/tmp/pip-build-env-sib7ewi4/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 320, in run_setup
          exec(code, locals())
        File "<string>", line 1, in <module>
        File "/tmp/pip-build-env-sib7ewi4/overlay/lib/python3.11/site-packages/setuptools/__init__.py", line 117, in setup
          return distutils.core.setup(**attrs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-sib7ewi4/overlay/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 160, in setup
          dist.parse_config_files()
        File "/tmp/pip-build-env-sib7ewi4/overlay/lib/python3.11/site-packages/setuptools/dist.py", line 652, in parse_config_files
          pyprojecttoml.apply_configuration(self, filename, ignore_option_errors)
        File "/tmp/pip-build-env-sib7ewi4/overlay/lib/python3.11/site-packages/setuptools/config/pyprojecttoml.py", line 72, in apply_configuration
          config = read_configuration(filepath, True, ignore_option_errors, dist)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-sib7ewi4/overlay/lib/python3.11/site-packages/setuptools/config/pyprojecttoml.py", line 140, in read_configuration
          validate(subset, filepath)
        File "/tmp/pip-build-env-sib7ewi4/overlay/lib/python3.11/site-packages/setuptools/config/pyprojecttoml.py", line 61, in validate
          raise ValueError(f"{error}\n{summary}") from None
      ValueError: invalid pyproject.toml config: `project`.
      configuration error: `project` must contain ['name'] properties
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

Is this the right way to install it? Do you recommend any other way?

[pcmoritz]

@gadregayatri Since you are just using the example, you don't actually need to install a customer version, just grab it from the current master docs, which are here https://docs.ray.io/en/master/serve/tutorials/vllm-example.html (I got there via the version selector in the top right of the docs).

If you really want to install an unreleased version of Ray (which is not needed), you can just use the method from the docs https://docs.ray.io/en/master/ray-overview/installation.html#installing-from-a-specific-commit and use an already compiled wheel :)

[power-puff-gg]

Hi @pcmoritz ,

Thank you pointing that out. I just used the updated example code.
I had to upgrade vllm version from 0.6.6.post1 to version: 0.7.2 as the vllm.entrypoints.openai.serving_models modules was not present in the aforementioned version.

I am not sure if this is related but earlier on 0.6.6.post1, vllm serve was at least able to launch a 'default' app using the below command:
CUDA_VISIBLE_DEVICES="0" serve run llm:build_app model="NousResearch/Meta-Llama-3-8B-Instruct" tensor-parallel-size=1 accelerator="GPU"

But after upgrading it to 0.7.2 I am getting the below GPU error though the GPU is free for usage:

(ServeController pid=3125831)   File "/root/miniconda3/envs/py311vLLMLatest/lib/python3.11/site-packages/torch/cuda/__init__.py", line 319, in _lazy_init
(ServeController pid=3125831)     torch._C._cuda_init()
(ServeController pid=3125831) RuntimeError: No CUDA GPUs are available

Not sure if it is connected to upgrading the package to 0.7.2. Any idea?

[devin-astrumu]

@gadregayatri were you able to fix this on 0.7.2? Encountering the same issue.

[johnny12150]

Having the same RuntimeError: No CUDA GPUs are available issue with vllm 0.7.3 and ray 2.42.1

(ServeController pid=644)   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker.py", line 155, in init_device
2025-02-22T17:43:11.389593881Z (ServeController pid=644)     torch.cuda.set_device(self.device)
2025-02-22T17:43:11.389596276Z (ServeController pid=644)   File "/usr/local/lib/python3.12/dist-packages/torch/cuda/__init__.py", line 478, in set_device
2025-02-22T17:43:11.389598760Z (ServeController pid=644)     torch._C._cuda_setDevice(device)
2025-02-22T17:43:11.389601125Z (ServeController pid=644)   File "/usr/local/lib/python3.12/dist-packages/torch/cuda/__init__.py", line 319, in _lazy_init
2025-02-22T17:43:11.389603549Z (ServeController pid=644)     torch._C._cuda_init()
2025-02-22T17:43:11.389605854Z (ServeController pid=644) RuntimeError: No CUDA GPUs are available

Found similar issues reported to vllm repo

• [Bug]: RuntimeError: No CUDA GPUs are available in transformers v4.48.0 or above when running Ray RLHF example vllm-project/vllm#13597

Downgrade transformers to 4.47.1 seems ineffective

[pteric]

I've encountered the same issue. Hope someone experienced can help take a look at this problem.