[Feature] Dynamic Lora Support in SGLang

[grahama1970]

Checklist

• 1. If the issue you raised is not a feature but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
• 2. Please use English, otherwise it will be closed.

Motivation

In SGLang, I would to dynamically apply domain-specific Lora adapters to smaller/local models. Normally, I use SGLang for inference. Recently, I've switched to Vllm which already has the ability to unload/load adaptors: https://docs.vllm.ai/en/latest/usage/lora.html
If this feature is already exists in SGlang, can you add an example in the documentation?

Related resources

https://docs.vllm.ai/en/latest/usage/lora.html

[zhaochenyang20]

@Ying1123 and @Wenyueh is on this. Thanks!

[yileld]

it easy to implement dynamic lora, just copy lora_manager.init_loras() as lora_manager.update_loras(), and add a judge
`

    if not hasattr(self, 'lora_modules'):
        self.lora_modules = []
    for module_name, module in self.get_target_modules():
        lora_module = self.set_lora_module(module_name, module)
        if lora_module:
            self.lora_modules.append(
                (module_name, lora_module)
            )

`
then use lora_manager.update_loras() before model_runner.lora_manager.prepare_lora_batch(ret).

[Fridge003]

Can this issue be solved with the answer by @yileld above? Is it necessary to open a PR to fix up this issue？@Ying1123

[grahama1970]

If it's more complex than the above solution, I'd love a coded explanation on how Loras are applied to the model, an, if multiple adaptors can be applied simultaneously, from different knowledge bases like finance and math adaptors.

[zhaochenyang20]

If it's more complex than the above solution, I'd love a coded explanation on how Loras are applied to the model, an, if multiple adaptors can be applied simultaneously, from different knowledge bases like finance and math adaptors.

@Fridge003

[Fridge003]

If it's more complex than the above solution, I'd love a coded explanation on how Loras are applied to the model, an, if multiple adaptors can be applied simultaneously, from different knowledge bases like finance and math adaptors.

After reading relevant codes, I feel the above solution is not enough to support the feature well because LoraManager instance has many attributes to maintain and update. I will try to come up with a better solution.

[zhaochenyang20]

@Fridge003 Thanks for pointing this out, and I hope to see your solutions!

[grahama1970]

Hi. Has there been any progress on Dynamically switching LorRas in SGLang? Thanks in advance

[zhaochenyang20]

@Fridge003

Hi. Has there been any progress on Dynamically switching LorRas in SGLang? Thanks in advance

[Fridge003]

Hi. Has there been any progress on Dynamically switching LorRas in SGLang? Thanks in advance

Hi, this feature is still under development and I have started a relevant PR draft #2891. The feature design is also described in this PR.

[zhaochenyang20]

Cool! @Fridge003

[github-actions]

This issue has been automatically closed due to inactivity. Please feel free to reopen it if needed.