[Misc]: running multiple vLLM instances on a single ray cluster

[gitlawr]

Anything you want to discuss about vllm.

When running the vLLM OpenAI server on a Ray cluster(with nodes A/B/C/D), I want to specify particular nodes(e.g., node A and B) for deployment, enabling better control over multiple vLLM instances within a single Ray cluster. Currently, it seems that Ray integration appears limited to specifying tp and pp.

Is supporting custom placement group a feasible option?

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[ruisearch42]

For a vLLM instance, it already can be instantiated with a Ray placement group. If you specify resources like "node:172.31.15.128: 1.0" for the PG, it should place the vLLM to the proper nodes.

[gitlawr]

Is it configurable when running vllm serve? Or it needs to be done by using the vllm engine in python

[github-actions]

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!