How to use Splitwise(from microsoft) in vllm?

[Lin-Qingyang-Alec]

Microsoft have claimed that ”Splitwise“ is supported in vLLM, see
https://www.microsoft.com/en-us/research/blog/splitwise-improves-gpu-usage-by-splitting-llm-inference-phases/

So how to use it in vLLM? I could not find keyword about ”Splitwise“.

[monuminu]

same here

[Lin-Qingyang-Alec]

@monuminu From your homepage, it appears that you work at Microsoft. Don’t you know where this part of the code can be obtained?😂

[monuminu]

Let me get the info :)

[Lin-Qingyang-Alec]

@monuminu Ok, please keep me posted if there are any updates.

[monuminu]

Code will be released in some days

[Lin-Qingyang-Alec]

Thank you, I am looking forward to the day when the code is available.

[kd303]

As per the blog, this seems implemented or part of vLLM

Our approach is now part of vLLM(opens in new tab) and can also be implemented with other frameworks.

[Lin-Qingyang-Alec]

@kd303 I have read this blog yet. But do you know how to use it in vllm? I can not find any keyword in vllm code.

[aashaka]

Thanks for the interest in Splitwise code. We have opened #2472 to track the open sourcing of our internal prototype.

[hmellor]

#2809

[github-actions]

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!