Closed
Description
🚀 The feature, motivation and pitch
For allowing Kv-aware routers / load-balancers to potentially target directly a dp group / worker. For example, see this PR Nvidia's dynamo:
ai-dynamo/dynamo#1181
Alternatives
No response
Additional context
No response
Before submitting a new issue...
- Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.