Can vLLM be deployed on the same GPU used for model training?

[Derekkk]

I am exploring whether vLLM can be deployed on the same GPU that is also used for model training. Specifically, assuming a small model (e.g., 3B) that only occupies a portion of GPU memory during inference, would it be possible to run vLLM inference while using the same GPU for training?

E.g., for a single-node setup with 8 GPUs:

• vLLM server runs on GPU0.
• Model training utilizes all 8 GPUs, including GPU0.

Would this setup be feasible? Thank you.

[edbeeching]

You can, but you would have to lower the vllm memory usage with --gpu_memory_utilization=0.5 which in turn would mean the memory available per device for training is not the same accross all devices. Which would mean a lower per_device_batch_size would have to be used.

I would run vllm on device 0 and training on devices 1,2,3,4,5,6,7 as I think that would achieve the best thoughput.

[qgallouedec]

You might encounter an NCCL error when creating the distributed group due to a duplicated GPU in the group (GPU 0). One user reported that it worked fine though (see huggingface/trl#3158).
If you manage to get it working, please share your experience. Like @edbeeching, I don’t think this is the ideal setup, but I’m still curious to see if it works.

[Derekkk]

Thanks for the reply. I did encounter the NCCL error while deploying vLLM on the GPU used in the group. I'm still working on a solution. Personally I feel like this setup would be useful when we have a limited number of GPU cards but sufficient memory for training a small model.

[qgallouedec]

Thanks for the feedback. There is an opened PR for this huggingface/trl#3162, if it can help