Support Kuberenetes for Distributed Serving

[sam-h-bean]

Only having support for ray for distributed inference will significantly reduce adoption of this tool if it truly is more performant than TGI. TGI can be run as a black-box image on Kubernetes with support for sharded models and vLLM should support this as well.

[hughesadam87]

Does this ticket mean that distributed serving is not supported in kubernetes, even if k8s has ray installed as per this quickstart guide?

https://docs.ray.io/en/latest/cluster/kubernetes/getting-started/raycluster-quick-start.html

The docs are very sparse here and I am confused, since they imply ray can be used for distributed inf.: https://vllm.readthedocs.io/en/latest/serving/distributed_serving.html

[github-actions]

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!

[github-actions]

This issue has been automatically closed due to inactivity. Please feel free to reopen if you feel it is still relevant. Thank you!