[Serve] Detailed Analysis of Errors Related to 'Ray does not allocate any GPUs on the driver node' && 'No CUDA GPUs are available'

[huiyeruzhou]

What happened + What you expected to happen

When deploying platforms based on the Ray framework, such as Ray Serve and Ray LLM, together with vLLM's OpenAI server, the errors "No CUDA GPUs are available" or "Ray does not allocate any GPUs on the driver node" have become recurring issues.
In this issue, I will provide a detailed analysis of these problems, along with a brief solution, experimental records. I sincerely invite developers from the Ray and vLLM communities to participate in the discussion, point out any shortcomings, and share your suggestions!

Quick Troubleshoot

For older versions of vLLM, I have also provided a hack to temporarily resolve this issue. Please refer to: Ray Issue #51154.

For Ray LLM and Ray Serve documentation:

• Ray LLM: Ray LLM Documentation
• Ray Serve: Ray Serve vLLM Example

A proper configuration for TP=1 involves modifying the build_app function in the example code from the Ray Serve documentation by replacing the following content.

    pg_resources = []
-    pg_resources.append({"CPU": 1})  # for the deployment replica
    for i in range(tp):
        pg_resources.append({"CPU": 1, accelerator: 1})  # for the vLLM actors

    # We use the "STRICT_PACK" strategy below to ensure all vLLM actors are placed on
    # the same Ray node.
    return VLLMDeployment.options(
+       ray_actor_options={"num_gpus": 1,"num_cpus": 1},
        placement_group_bundles=pg_resources, placement_group_strategy="STRICT_PACK"
    ).bind(

---

Introduction

The issue can be summarized simply: the framework design of vLLM does not fully accommodate LLMEngine running within a placement group.

The process that creates RayDistributedExecutor, which serves as the entry point, must have access to a GPU while not occupying GPU resources within Ray. This conflicts with the typical configuration of Ray Serve. Additionally, since vLLM always requests a whole number of GPUs when world_size > 1, it is not possible to work around this limitation by allocating fractional GPUs.

Regardless of whether using LLM (offline inference) or OpenAIServingCompletion (online deployment), both are considered entry points. The class responsible for managing the specific processes during initialization is called an Executor. The Executor itself creates a local actor to use the GPU and also spawns a dummy actor to reserve resources in the placement group.

However, when integrating this framework into Ray, several issues arise:

• In Ray, the Executor itself also runs within an Actor and uses the first bundle of the placement group.
  • If no GPU resources are assigned to it, CUDA_VISIBLE_DEVICES will be an empty string, leading to the "No CUDA GPUs are available" error when trying to call set_device.
  • On the other hand, if we do allocate a GPU to it, vLLM will still use a dummy_driver_worker that occupies a GPU, which causes the total number of requested workers to exceed the placement group capacity.
  • Since vLLM does not allocate resources based on bundles but instead forces each worker to use exactly one GPU when world_size > 1, we cannot work around this limitation by assigning fractional GPUs.

A Deadlock!

---

Experiments

Due to the specific feature of the code, there are actually two executable scenarios. I will first present the experimental table and then analyze each case one by one.

VLLM Version	Placement Group Configuration	TP	Status	Notes
VLLM 0.7.3	[{'CPU':1} + {'GPU':1} * TP]	>1	✅ Works	Replica actor has no GPU but gains access via update_environment_variables
VLLM 0.7.3	[{'GPU':1} * TP]	>1	❌ Fails	Extra worker creation causes deadlock due to loop in ray_distributed_executor.py#L187
VLLM 0.7.3	[{'CPU':1} + {'GPU':1} * TP]	1	❌ Fails	Replica actor has no GPU, and Executor can no longer "borrow" CUDA_VISIBLE_DEVICES
VLLM 0.7.3	[{'GPU':1} * TP]	1	✅ Works	Replica actor has no GPU, but uniproc_executor avoids dummy worker creation

---

Analysis

In the existing code, there are actually two scenarios where execution is possible:

1. TP > 1 without explicitly assigning GPUs (this is the default setting in Ray Serve). This explains why the issue has not become a critical blocker—under the current configuration, execution is still possible.
2. TP = 1 with GPU assignment (as mentioned earlier, using an appropriate configuration combined with Ray Serve to resolve the issue).

Case 1: Default Configuration (TP > 1 & No GPU Assigned)

Even if Ray does not allocate any GPUs to the Replica Actor (i.e., the RayDistributedExecutor within the Serve framework), CUDA_VISIBLE_DEVICES will still not be empty.

This happens because of this line of code, which calls self.driver_worker and modifies the environment variables of the current process.

As a result, in the default configuration, the code functions correctly, allowing a process to access GPUs without directly occupying them.

Case 2: TP = 1 Changes the Behavior

When TP = 1, vLLM switches to using UniprocExecutor, as seen in this line of code.

In this case, if CUDA_VISIBLE_DEVICES is empty, it will cause an error, as UniprocExecutor does not inherit the same environment variable handling as the multi-process setup.

---

Supplementary Notes on Ray Serve and Ray LLM

After an initial review of the source code and conducting simple experiments, I believe that the new and old APIs of Ray Serve are fundamentally the same, except for the addition of a router and deeper integration with vLLM.

The core interaction between Ray and vLLM still revolves around the placement group (PG) allocation during deployment.

Therefore, these two approaches are essentially equivalent:

1. Manually integrating vllm.entrypoints.openai.serving_completion into Ray Serve.
2. Using the ray[llm] library for deployment.

---

Related Issues

Based on my preliminary review, the following issues are all related to the analysis presented here:

• vLLM Issue #12983
• vLLM Issue #13521
• vLLM Issue #14415
• vLLM Issue #14456
• Ray Issue #51154
• Ray Issue #51193
• Ray Issue #50275

Versions / Dependencies

vllm>=0.7.2
ray[serve,llm,default] -U

Reproduction script

Demo code in following

• Ray LLM: Ray LLM Documentation
• Ray Serve: Ray Serve vLLM Example

Issue Severity

High: It blocks me from completing my task.

[kouroshHakha]

Hi, so I did some digging and here is the explanation for your observation and ultimately some suggested workarounds:

The reason that TP=1 breaks with the existing example code is that vllm will fall back to UniProcExecutor when TP=1. This executor will need the resources to be provisioned at the serve replica, which you already noted. To fix this using the same pattern as TP > 1, you can force vllm to use ray distributed backend by passing engine_args.distributed_executor_backend = "ray".

After an initial review of the source code and conducting simple experiments, I believe that the new and old APIs of Ray Serve are fundamentally the same, except for the addition of a router and deeper integration with vLLM.
The core interaction between Ray and vLLM still revolves around the placement group (PG) allocation during deployment. Therefore, these two approaches are essentially equivalent

This is not entirely true. In the serve llm apis, we are using ray distributed backend regardless of TP. This is the main reason why the serve api will work out of the box in both TP=1 and TP>1 context. I highly recommend switching over to that instead of using the existing vllm serve example (which we'll soon remove to reduce confusion of entry path now that we have serve llm apis). There are more subtle details for vllm-v1 which we will handle as part of our data and serve llm APIs so the ray + vllm deployments will work out of the box.

[huiyeruzhou]

Thanks for your correction! You’re absolutely right—the executor type is indeed a key consideration. However, if we have a Ray cluster with a CPU-only head node and additional worker nodes equipped with GPUs, the replica actor might be scheduled on the head node, leading to the same issue. That’s exactly when I first encountered the "No CUDA GPUs" error.

Maybe we should add "accelerator" resource spec to the first bundle in PG to ensure the replica actor is scheduled to a GPU node.

Additionally, I noticed that up until vLLM 0.7.3, the LLM engine enforces the creation of a uniproc executor when TP=1, even if the backend is explicitly set. This behavior is discussed in this PR.

As for why I have to use Ray Serve—my GPU platform runs on a heavily modified version of Ray to support an IPv6 network within our Kubernetes cluster. Because of this, I’m constrained to using a lower version of ray provied by my platform in multi-node task 😂. I still want to find a workaround to make it run if it is possible!

While the expriment result shown here is running in latest ray, with a single node.

[kouroshHakha]

Thanks for your correction! You’re absolutely right—the executor type is indeed a key consideration. However, if we have a Ray cluster with a CPU-only head node and additional worker nodes equipped with GPUs, the replica actor might be scheduled on the head node, leading to the same issue. That’s exactly when I first encountered the "No CUDA GPUs" error.

If you use STRICT_PACK (or even PACK) for the placement-group, both replica and all the vllm workers will be forced to be on the same locality which means the serve replica should not get scheduled on the head node if possible (docs). In case of PACK it will only do it on a different node when the resources on a single node are not satisfied.

Additionally, I noticed that up until vLLM 0.7.3, the LLM engine enforces the creation of a uniproc executor when TP=1, even if the backend is explicitly set. This behavior is discussed in vllm-project/vllm#12934.

Yes this was a bug that was breaking our solution as well. It should be fixed after that PR.

As for why I have to use Ray Serve—my GPU platform runs on a heavily modified version of Ray to support an IPv6 network within our Kubernetes cluster. Because of this, I’m constrained to using a lower version of ray provied by my platform in multi-node task 😂. I still want to find a workaround to make it run if it is possible!

Understood why ray upgrade is not possible. Push for an upgrade :) We have a lot of awesome stuff cooking on both the data and serve llm stack which you might not want to recreate. Roadmap will get posted soon.

[huiyeruzhou]

Thanks for your clarification again, sorry for forgetting placement strategy：) I think ray llm is a great api and it always works well with vllm, the issue mentioned above does not effect it since ray llm directly call the constructor instead of from_engine_args

Here is also a new some finding about ray serve. (I have to trace every function call to troch.cuda to locate it....)
The current RayDistributedExecutor relis on resetting CUDA_VISIBLE_DEVICES, which means torch._C._cuda_getDeviceCount() should not be called before the engine is built, since the function always returns the same value based as the first call. This is because of the static vaiable used in Cpp source code.

My envrionment somehow depends on tensordict library, which will call torch.cuda.is_available() when it is imported, and therefore the reset of CUDA_VISIBLE_DEVICES will be ignored : (
torch.cuda.is_available(), torch.cuda.device_count() are 2 functions that will freeze the effect of CUDA_VISIBLE_DEVICES, there maybe more function in _C that has this side effect.

Adding os.environ['CUDA_VISIBLE_DEVICES'] = '0,1,2,3,4,5,6,7' in the __init__ of VLLMDeployment class can make it work well with RayDistributedExecutor in almost any environment. It may have some side effects, but the mechanism is clear now.
Another way is to specify {‘GPU': tp} and use MultiprocessingDistributedExecutor instead， which means ray can free from holding GPU for vllm's executor.
I think this two ways can be a final solution of ray serve for VLLM > 0.7.0.
For VLLM <= 0.7.2, we must specify the class, "ray" will be ignored when TP =1

Here for code implementation

# RayDistributedExecutor
@serve.deployment(
    num_replicas=1,
    max_ongoing_requests=128,
)
@serve.ingress(app)
class VLLMDeployment:
    def __init__(
...
):
	# in the begnning of init function
	import os
	os.environ['CUDA_VISIBLE_DEVICES'] = '0,1,2,3,4,5,6,7'
	...
	from vllm.executor.ray_distributed_executor import RayDistributedExecutor
	engine_args.distributed_executor_backend = RayDistributedExecutor # "ray" is not enough for tp = 1
	self.engine = AsyncLLMEngine.from_engine_args(engine_args)

...
def build_app(cli_args: Dict[str, str]) -> serve.Application:
	...
	pg_resources=[{"CPU": 1}] + [{"GPU": 1}] * tp
	return VLLMDeployment.options(
	    placement_group_bundles=pg_resources,
	    placement_group_strategy="STRICT_PACK",
	).bind(
	...

# MultiprocessingDistributedExecutor, always work
@serve.deployment(
    num_replicas=1,
    max_ongoing_requests=128,
)
@serve.ingress(app)
class VLLMDeployment:
    def __init__(
...
):
	from vllm.executor.mp_distributed_executor import MultiprocessingDistributedExecutor
	engine_args.distributed_executor_backend = MultiprocessingDistributedExecutor
	self.engine = AsyncLLMEngine.from_engine_args(engine_args)
	...
def build_app(cli_args: Dict[str, str]) -> serve.Application:
	...
	pg_resources=[{"CPU": 1, "GPU": tp}]
	return VLLMDeployment.options(
	    ray_actor_options={"num_gpus": tp,"num_cpus": 1,},
	    placement_group_bundles=pg_resources,
	    placement_group_strategy="STRICT_PACK",
	).bind(
	...

[kouroshHakha]

@huiyeruzhou I think a better compromise instead of setting os.environ['CUDA_VISIBLE_DEVICES'] = '0,1,2,3,4,5,6,7' in VLLMDeployment would be to use

pg_resources=[{"CPU": 1, "GPU": 1} for _ in range(tp)]

In other words you bundle CPU and GPUs together. This way the replica would still claim a CPU from one of these bundles which come with a GPU. Therefore ray would not unset the CUDA_VISIBLE_DEVICES within the context of VLLMDeployment. So I suppose this should fix the caching behavior you are seeing with NO CUDA DEVICES found?

The key idea is the relationship between actors to bundles is many-to-one. So this could work ^

[huiyeruzhou]

Hello! I apologize for the delayed response.

pg_resources = [{"CPU": 1, "GPU": 1} for _ in range(tp)]

The central idea here is that there's a many - to - one relationship between actors and bundles. Thus, this code should work as expected. ^

This approach holds true in most common scenarios. However, vllm is a rather special case. It creates a dummy worker to handle resource management for the driver - worker. If the driver has already been allocated a GPU, the creation of this extra worker will lead to the code getting stuck.