[Misc] Enable vLLM to Dynamically Load LoRA from a Remote Server

[angkywilliam]

In the current implementation, vLLM only supports loading LoRA from local storage.

At OpenPipe, we are extending the serving engine's capabilities by introducing a LoRAResolver. LoRAResolver enables vLLM users to implement custom logic for fetching LoRA files from remote servers.

For example, in OpenPipe's case, we are dynamically loading LoRA for our customer from S3.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

• Add ready label to the PR
• Enable auto-merge.

🚀

[angkywilliam]

@simon-mo @khluu can I get help to unblock buildkite, thanks!

[khluu]

@simon-mo @khluu can I get help to unblock buildkite, thanks!

which test do you need to unblock? of if you want permission to unblock tests, can I have your email associated with your github?

[angkywilliam]

@simon-mo @khluu can I get help to unblock buildkite, thanks!

which test do you need to unblock? of if you want permission to unblock tests, can I have your email associated with your github?

Hey @khluu , this is my first time opening PR to vLLM.

Do I need to pass all the test in BuildKite for the code to be reviewed and merged?
If yes, I would need help to unblock all the test in BuildKite.

I also send you email with subject Unblock Dynamically Load LoRA from a Remote Server for my email address.

Thanks for the help!

[jeejeelee]

IIUC, although this PR is related to LoRA loading, it seems you haven't touched the underlying LORA logic. What you might need is to add unit tests similar to #6566.
BTW, maybe you can refer to : https://github.com/vllm-project/vllm/blob/main/vllm/lora/worker_manager.py#L94

[khluu]

@simon-mo @khluu can I get help to unblock buildkite, thanks!

which test do you need to unblock? of if you want permission to unblock tests, can I have your email associated with your github?

Hey @khluu , this is my first time opening PR to vLLM.

Do I need to pass all the test in BuildKite for the code to be reviewed and merged? If yes, I would need help to unblock all the test in BuildKite.

I also send you email with subject Unblock Dynamically Load LoRA from a Remote Server for my email address.

Thanks for the help!

Oh if it's just for merging, you can just wait until your PR is reviewed and approved, then your PR reviewer can run CI for you. Fastcheck is more like a subset of CI, running short tests and give you the flexibility to run extra specific test if needed (which you need to be in Buildkite org to do).

[DarkLight1337]

@youkaichao do you think we can make this a plugin?

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @angkywilliam.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[corbt]

I would love to see this merged in!

[youkaichao]

For LoRA specific code, @jeejeelee has more context than me to review the code.

From the integration perspective, I think it's okay to create a lora resolver registry, and expose a function to register new lora resolvers. We don't need a new plugin category. General plugin should be enough to call the function to register new lora resolvers.

One technical detail is, we should collect all registered resolvers, and try them one by one to see which resolver succeeds. We don't need to let users select the resolver.

[angkywilliam]

Thanks for the feedback @youkaichao

From what I understand so far, there are three main areas I'd need to modify for LoRA plugin support:

1. lora/resolver.py

• Introduce an abstraction for LoRAResolver
• Add a LoraResolver registry to enable plugin registration

2. entrypoints/openai/serving_engine.py

• Update _check_model to iterate through the registered lora_resolvers after the lora_requests check

3. Register the custom lora_resolver

• This registration would happen outside of vLLM codebase, correct?

Does this sounds like a reasonable approach @youkaichao @jeejeelee?
Want to make sure I have the right direction, before implementing it.

Thanks in advance!

[jeejeelee]

@joerunde WDYT

[angkywilliam]

I’ve made the concrete change here: #15733
Let me know if this make sense!

[angkywilliam]

Updated the code with:

• Merge latest code from main
• Refactor lora resolver as a plugin per @youkaichao's input

[angkywilliam]

@joerunde I’ve added the test for LoRA loading flow via LoRAResolver, along with the corresponding documentation.
I believe the code is ready to be merged — let me know if you have any additional feedback.

[joerunde]

LGTM!

Thanks for the plugin support, @jeejeelee @youkaichao what do y'all think about getting this merged?

[simon-mo]

Can you add an example in the documentation for such plugin? so we have some examples people can leverage and knowing how to use it.

[angkywilliam]

@simon-mo I’ve added an example of the LoRAResolver implementation to the documentation

[jberkhahn]

also if you want to see what a resolver implementation would look like, i have my PR converted to a plugin up here: https://github.com/jberkhahn/filesystem_resolver

[PedroMiolaSilva]

@angkywilliam hey! I've been trying to use LoRA resolver on local file system with no luck. I've detailed my attempt here: #18630. Any help is welcome :)

After it works locally, I want to try loading from s3.