Skip to content

Conversation

dtcccc
Copy link
Contributor

@dtcccc dtcccc commented Sep 19, 2025

See vllm-project/vllm#24718

Support mooncake transfer engine in vllm v1 by using dynamic external kv_connector.

Example usage:

Adding the following params to vllm command:
Prefill: --kv-transfer-config '{"kv_connector":"MooncakeConnector","kv_role":"kv_producer", "kv_connector_module_path":"mooncake.mooncake_connector_v1"}'
Decode: --kv-transfer-config '{"kv_connector":"MooncakeConnector","kv_role":"kv_consumer", "kv_connector_module_path":"mooncake.mooncake_connector_v1"}'
Proxy: Running tests/v1/kv_connector/nixl_integration/toy_proxy_server.py

@dtcccc dtcccc changed the title [Transfer Engine] feat: Support external kv_connector for vllm v1 [Misc] feat: Support external kv_connector for vllm v1 Sep 19, 2025
Signed-off-by: Tianchen Ding <dtcccc@linux.alibaba.com>
Copy link
Collaborator

@ShangmingCai ShangmingCai left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thx for the contribution! LGTM. Before we merge it, can you add a demo or a README on how to use it?

Signed-off-by: Tianchen Ding <dtcccc@linux.alibaba.com>
@dtcccc
Copy link
Contributor Author

dtcccc commented Sep 19, 2025

Thx for the contribution! LGTM. Before we merge it, can you add a demo or a README on how to use it?

Added a demo in README.

@stmatengss
Copy link
Collaborator

Could you integrate a proxy server from vllm rep? So we don't rely on vllm proxy server

Copy link
Collaborator

@ShangmingCai ShangmingCai left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Optimizations and proxy can be placed in the future PR. Thx for the contribution.

@ShangmingCai ShangmingCai merged commit a97223b into kvcache-ai:main Sep 23, 2025
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants