Fix num_gpus when TP > 1

[WoosukKwon]

Fixes #1851

[WoosukKwon]

@Yard1 Should we also fix this:

vllm/vllm/engine/async_llm_engine.py

Line 304 in f07c1ce

engine_class = ray.remote(num_gpus=1)(self._engine_class).remote

?

[Yard1]

Yes, that line in AsyncEngine it should be set in the same way @WoosukKwon

[WoosukKwon]

I see. It seems tricky then...

[Yard1]

Perhaps make it a method of ParallelConfig to get the number of GPUs for Ray?

[WoosukKwon]

Perhaps make it a method of ParallelConfig to get the number of GPUs for Ray?

@Yard1 Can we somehow keep ParallelConfig independent of Ray?

[Yard1]

Have an independent function taking in the parallel config, then?

[FlorianJoncour]

In my opinion its a regression compared to #1821

After further investigations it seems that Ray make CUDA fail with tensor parallelism if the total amount of GPU reservation is less than 1 (wich indeed no make sense).

Should we not raise and exception in that case ?

# Ensures that we reserves at least 1 GPU with tensor parallelism
if self.parallel_config.tensor_parallel_size > 1:
    gpu_res = (self.cache_config.gpu_memory_utilization * self.parallel_config.tensor_parallel_size)
    if gpu_res < (self.parallel_config.tensor_parallel_size-1):
        raise ValueError(f"`tensor_parallel_size` is set to {self.parallel_config.tensor_parallel_size} but the current placement group bundle reserves less than {self.parallel_config.tensor_parallel_size} GPU ({gpu_res}). "
                            f"Try increasing `gpu_memory_utilization` to at least 0.5. "
                            f"If you are using Ray Serve you may also need to set a correct `placement_group_bundles`")

And let

num_gpus=self.cache_config.gpu_memory_utilization

So it helps the user to set correct configuration and also let it use fractions of GPU with tensor parallelism.
I've tested with and without Ray Serve and we just have to set correct gpu_memory_utilization
And also placement_group_bundles with Ray serve.

[Yard1]

So the problem here is that NCCL requires separate devices to operate, but Ray has no insight into that and will try to pack the placement group into as few GPUs as possible. If the number of GPUs Ray chooses is smaller than the tensor parallelism factor, NCCL will not work. There is no easy way to prevent that aside from just not using fractional GPUs if tensor parallelism factor is greater than 1.

[WoosukKwon]

@Yard1 I'ved fixed async_llm_engine. While the fix is a bit hacky, I think this is acceptable since engine_use_ray is usually False. PTAL.