Multi-LoRA - Support for providing /load and /unload API

[gauravkr2108]

Problem statement:

In the production system, there should be an API to add\\remove fine-tuned weights dynamically. Inference caller should not have to specify LoRA location with each call.

Current Multi-LoRA support allows adaptor load during inference calls, which doesn't check if finetune weights are already loaded and ready for inferencing.

Proposal:

Introduce an API - /load and /unload to allow fine-tuned weights inclusions in vllm.

POST /load -> add finetunes weight as part of models.
POST /unload -> remove finetunes weight from models list.

This will allow the set of finetuned weights present in vllm server.

This will infer no need to specify finetune weight names, and locations as part of each inference request.

Sample code:

lora_request = None
index = 1
 
 
@app.post("/load")
async def load(request: Request) -> Response:
    request_dict = await request.json()
    global lora_request
 
    lora_local_path = request_dict.pop("lora_path", "/models/lora/")
    global index
    lora_request = LoRARequest(
        lora_name=lora_local_path,
        lora_int_id=index,
        lora_local_path=lora_local_path)
 
    index = index + 1
    return Response(status_code=201)
 
@app.post("/unload")
async def unload(request: Request) -> Response:
    """
    Unload API
    :param request:
    :return:
    """
    global lora_request
    lora_request = None
 
    global index
    if not index <= 1:
        index = index - 1
 
    return Response(status_code=201)

[simon-mo]

I'm open to this because I anticipate this will be helpful for production use cases. PRs are welcomed with changes to OpenAI API server with route starting with /-/ indicating private API. Such as PUT /-/lora_cache and DELETE /-/lora_cache.

[gauravkr2108]

let me draft a PR

[simon-mo]

I have seen #3446 pop up

[gauravkr2108]

@simon-mo add for add and delete operation; I can work with #3446 to add delete operation.

[gauravkr2108]

@simon-mo there is an Open AI API to delete fine-tuned model https://platform.openai.com/docs/api-reference/models/delete ; we adopt this API?

[thincal]

So that if I want to inference an LoRA model, needs to invoke the /load firstly then inference with lora model? if there are many vLLM engine instance deployed, the requests is load-balanced to selected vLLM instance, how does this two step design work to make sure inference happened in an instance with the lora just loaded already?

[lizzzcai]

created one feature request but for multi models, I think the endpoint can be reused.

[DarkLight1337]

Closing as completed by #6566