[Usage]: Pass multiple LoRA modules through YAML config

[andreapairon]

How would you like to use vllm

I would like to pass multiple LoRA modules to the vLLM engine, but currently I'm receiving error while parsing the lora_modules property.

The LoRAParserAction class receives a Sequence[str] in case you want to use multiple LoRA modules.

I have a YAML config file in which I declare the vLLM engine arguments, like this:

model: ai-models/Meta-Llama-3.1-8B-Instruct-rev-5206a32
tokenizer_mode: auto
dtype: half
lora_modules: "ai-models/adv_perizia_exp7_run6=ai-models/adv_perizia_exp7_run6"
max_num_batched_tokens: 32768
max_num_seqs: 192
gpu_memory_utilization: 0.95
tensor_parallel_size: <RAY_LLM_NUM_WORKERS>
max_model_len: 32768

In that way (name=path for the LoRA module), all works and I'm able to perform inference with LoRA (I set enable_lora argument later in the code, not in the YAML file).
Now I would like to pass multiple lora_modules, but I'm receiving parsing error in every different ways I tried:

lora_modules: "ai-models/adv_perizia_exp7_run6=ai-models/adv_perizia_exp7_run6 ai-models/perizia_exp7_run3=ai-models/perizia_exp7_run3" (blanks space between LoRA modules)

lora_modules:
      - "ai-models/adv_perizia_exp7_run6=ai-models/adv_perizia_exp7_run6"
      - "ai-models/perizia_exp7_run3=ai-models/perizia_exp7_run3"

lora_modules: ["ai-models/adv_perizia_exp7_run6=ai-models/adv_perizia_exp7_run6","ai-models/perizia_exp7_run3=ai-models/perizia_exp7_run3"]

lora_modules: > "ai-models/adv_perizia_exp7_run6=ai-models/adv_perizia_exp7_run6\nai-models/perizia_exp7_run3=ai-models/perizia_exp7_run3" (\n between LoRA modules)

How can I pass multiple LoRA modules correctly? Thanks in advance.

[jeejeelee]

See: https://docs.vllm.ai/en/latest/models/lora.html#serving-lora-adapters

[andreapairon]

@jeejeelee I saw that, but passing multiple LoRA by using a blank space --lora-modules {name}={path} {name}={path} it's not working in the YAML.

I tried:

lora_modules: "ai-models/adv_perizia_exp7_run6=ai-models/adv_perizia_exp7_run6 ai-models/perizia_exp7_run3=ai-models/perizia_exp7_run3" in which the second LoRA module (ai-models/perizia_exp7_run3=ai-models/perizia_exp7_run3) is separated from the first one by using a blank space.

I receive that stack trace:

Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/ray/serve/_private/application_state.py", line 1049, in build_serve_application
 app = call_app_builder_with_args_if_necessary(import_attr(import_path), args)
File "/usr/local/lib/python3.10/dist-packages/ray/serve/_private/api.py", line 324, in call_app_builder_with_args_if_necessary
 app = builder(args)
File "/serve_app/src/main.py", line 34, in app_builder
 parsed_args = parse_vllm_args(engine_config["engine_kwargs"])
File "/serve_app/src/config/config_parser.py", line 30, in parse_vllm_args
 parsed_args = parser.parse_args(args=arg_strings)
File "/usr/local/lib/python3.10/dist-packages/vllm/utils.py", line 1114, in parse_args
 return super().parse_args(processed_args, namespace)
File "/usr/lib/python3.10/argparse.py", line 1845, in parse_args
 args, argv = self.parse_known_args(args, namespace)

[jeejeelee]

If you have multiple LoRAs, YAML passing is currently not supported. As a temporary solution, you can append args like:

--config  your.yaml --lora_modules lora0=path0 lora1=path1 

[andreapairon]

Ok, thank you for your answer!

[jeejeelee]

BTW, Is passing lora_modules through YAML a common method? If it is commonly used, I can help support this feature.

[andreapairon]

I don't know if it's a common practice, but we currently have a YAML file with engine configs and args that in fact is a Kubernetes config map. Then we mount and read this config map as a Dict and pass them to the arguments parser of vLLM, which, for example, substitute _ with - and add -- at the beginning of the argument name to make the arguments "vLLM-cli compatible".

[jeejeelee]

If that's the case, you can also consider converting the lora_modules in the YAML to a format supported by vLLM. In the YAML, you can write it in list or str type

[andreapairon]

Yeah, I tried as I wrote in my question, but something seems going wrong during the parsing.
I tried as a YAML list:

lora_modules:
      - "ai-models/adv_perizia_exp7_run6=ai-models/adv_perizia_exp7_run6"
      - "ai-models/perizia_exp7_run3=ai-models/perizia_exp7_run3"

[jeejeelee]

If you using --config your.yaml, vllm will convert YAML list to str, see: https://github.com/vllm-project/vllm/blob/main/vllm/utils.py#L1260

[andreapairon]

Uh ok, that is useful. Thanks!