Support multi-node inferencing using LWS

[sriumcp]

Is your feature request related to a problem? Please describe.

ModelService currently supports disaggregated prefill and decode deployments, including tensor-parallel setups across multiple GPUs on a single node. However, to enable multi-node vLLM inference, we require integration with the leaderworkerset controller.

For example, consider the following use-case: I want to run vLLM with tensor_parallel_size=8, distributed across 2 nodes with 8 GPUs each. For this use-case, I would like ModelService to support orchestrating a multi-pod workload under a LeaderWorkerSet, where each pod (worker) forms part of a single logical model instance.

Describe the solution approach you'd like

We need native support in ModelService controller for:

Creating and managing LeaderWorkerSet CRs.

Propagating relevant vLLM CLI/env args to each pod based on its role (leader vs worker).

Handling lifecycle and readiness conditions for a distributed setup.

Reusing existing prefill/decode fields as much as possible, perhaps adding a new field like useLeaderWorkerSet: true.

Complete the implementation of the parallelism stanza in ModelService (including node parallelism)

Describe alternatives you've considered

Manually creating LeaderWorkerSet CRs and bypassing ModelService, which breaks the declarative, unified model serving approach.

Additional context
See:

https://github.com/kubernetes-sigs/lws

https://docs.vllm.ai/en/latest/deployment/frameworks/lws.html

The goal is to make ModelService a first-class abstraction for both single-node and multi-node vLLM deployments.

📌 Please read the linked references carefully before implementing this feature.

[mrunalp]

Dupe of #76

[sriumcp]

Thanks @mrunalp. If it's all the same, let's please track here. We should start by adding a few more design details here.

---

Comments from the original issue below.

https://docs.vllm.ai/en/latest/deployment/frameworks/lws.html

Also, an operator is in progress for OCP: https://github.com/openshift/lws-operator

(Opening this for tracking. This isn't something that needs to be implemented initially.)

[mrunalp]

Sounds good. Let's close #76. Thanks!

[mrunalp]

@ardaguclu fyi

[sriumcp]

We want to create deepseek samples.

From @ljfh001 ...

For deepseek, we need to provide reference LWS yamls for different hardware(e.g. H100, H200, B200) for both IB and RoCE

[nicole-lihui]

Hi， @sriumcp @asm582 is there anything I can help with on this PR?

I'm really interested in this feature, and if possible, I’d love to get involved. I have some hands-on experience with LWS and a understanding of how it works.

Here’s my personal take on how this could be implemented:

1. the ModelService controller to support LWS as a new workload type (ideally reusing the existing Deployment interface with some abstraction to plug in different workload types like LWS).
2. Define different lws samples, such as deepseek-h100, deepseek-h200, etc
3. Add LWS usage documentation to the llm-d repo, and sync it with llm-d-deployer.

[nicole-lihui]

On LWS Support for P/D

Currently, LWS doesn’t have a production-ready solution for clean PD separation. But the commonly accepted approach in the community is to run two LWS instances—one for prefill and one for decode. A more native solution may come in the future. We’re tracking progress via LWS Issue #388.

There are a few existing community approaches to PD separation using LWS, each with its pros and cons. For example, as shared in this hk KubeCon More -- Model Sharding: LWS & Distributed Inference

1. A single LWS in kv both mode
2. A single LWS with kv diff, using a proxy-style leaderTemplate (like EPP), and splitting workers into prefill/decode by index
3. Two separate LWS instances, each with its own leaderTemplate – one for prefill, one for decode

Since PD separation is quite complex, we suggest starting simple:

Just reuse the existing Deployment-style architecture and run two LWS workloads to support larger models like DeepSeek. This also allows us to scale XP and XD independently.

Later on, we can adopt more advanced multi-node LWS setups once they stabilize—without needing major changes to either LWS or llm-d.

[ahg-g]

I think deploying two LWS to represent the decode and prefill deployments is actually a reasonable pattern. What is the benefit of coupling them into one?

[nicole-lihui]

I think deploying two LWS to represent the decode and prefill deployments is actually a reasonable pattern. What is the benefit of coupling them into one?

@ahg-g this is just my personal understanding.

P, D are two components that can only from a functioning LLM application when combined; they cannot exist independently.
they should share lifecycle management and other controls, unliker other kv-cache (optional). if there is a single workload CRD that can manage both together, that would definitely be a better choice. but current LWS not supported.

[sriumcp]

Not quite a single workload CRD, but the intent behind the Model Service Helm chart is that there's a single chart to manage both the P&D instances through a single install.

I know the Kserve community (cc: @terrytangyuan ) is creating a single CRD which can orchestrate LWS under the covers. Might be good to know the PD support from the KServe LLMInferenceService CRD.

[alexeldeib]

deploying two LWS to represent the decode and prefill deployments is actually a reasonable pattern

curious, without additional CRDs would you see it looking something like the draft POC that was opened before, just hardcoding LWS as an alternative to deployment and parsing it out of a config map? or should we generalize the internals a bit to make it easier to pivot on future resource types?

#219 the poc I mentioned

I managed to get some basic functionality working with further diffs on top of that branch, quite a bit is still broken like status updates right now. Curious if that's an angle folks would be interested to pursue, or go a different angle.