Better integration with Ray Serve

[FlorianJoncour]

I use several models using Ray Serve outside of Vllm.
Vllm ask for all ressources to Ray, making it impossible to use other models alongside.

So I use the gpu_memory_utilization parameter to limit the gpu ressources requested by the worker, which then allows placement_group_bundles to be used in Ray Serve deployments.

The RayWorker class has also been renamed to RayWorkerVllm to avoid ambiguities with other Ray actors.

Edit: Clarification, Vllm don't use all Vram, but requests for all ressources to Ray.

[matt-psaltis]

I think this feature fixes the underlying cause for the problem here: ray-project/ray-llm#94

Thanks @FlorianJoncour!

[FlorianJoncour]

Yes it seems to be the same issue.

I almost went crazy trying to get this to work before diving into the vllm code

[Yard1]

This looks good to me, though I would say that the rename is unnecessary :)

One important thing to note is this doesn't actually provide a hard boundary on the process memory usage, so using fractional GPUs doesn't have a 100% guarantee that OOMs will be avoided. That being said, they should be quite unlikely thanks to vLLM memory profiling.

[WoosukKwon]

@FlorianJoncour @Yard1 It seems this change causes a bug when gpu_memory_utilzation < 0.5 and tensor_parallel_size > 1:

RuntimeError: CUDA error: invalid device ordinal

I guess this is because num_gpus is set to gpu_memory_utilization? Do you have any idea to fix this?

[Yard1]

@WoosukKwon I see. For now we should do:

num_gpus=self.cache_config.gpu_memory_utilization if self.parallel_config.tensor_parallel_size < 2 else 1