[Usage]: Pass multiple LoRA modules through YAML config

[andreapairon]

How would you like to use vllm

I would like to pass multiple LoRA modules to the vLLM engine, but currently I'm receiving error while parsing the lora_modules property.

The LoRAParserAction class receives a Sequence[str] in case you want to use multiple LoRA modules.

I have a YAML config file in which I declare the vLLM engine arguments, like this:

model: ai-models/Meta-Llama-3.1-8B-Instruct-rev-5206a32
tokenizer_mode: auto
dtype: half
lora_modules: "ai-models/adv_perizia_exp7_run6=ai-models/adv_perizia_exp7_run6"
max_num_batched_tokens: 32768
max_num_seqs: 192
gpu_memory_utilization: 0.95
tensor_parallel_size: <RAY_LLM_NUM_WORKERS>
max_model_len: 32768

In that way (name=path for the LoRA module), all works and I'm able to perform inference with LoRA (I set enable_lora argument later in the code, not in the YAML file).
Now I would like to pass multiple lora_modules, but I'm receiving parsing error in every different ways I tried:

lora_modules: "ai-models/adv_perizia_exp7_run6=ai-models/adv_perizia_exp7_run6 ai-models/perizia_exp7_run3=ai-models/perizia_exp7_run3" (blanks space between LoRA modules)

lora_modules:
      - "ai-models/adv_perizia_exp7_run6=ai-models/adv_perizia_exp7_run6"
      - "ai-models/perizia_exp7_run3=ai-models/perizia_exp7_run3"

lora_modules: ["ai-models/adv_perizia_exp7_run6=ai-models/adv_perizia_exp7_run6","ai-models/perizia_exp7_run3=ai-models/perizia_exp7_run3"]

lora_modules: > "ai-models/adv_perizia_exp7_run6=ai-models/adv_perizia_exp7_run6\nai-models/perizia_exp7_run3=ai-models/perizia_exp7_run3" (\n between LoRA modules)

How can I pass multiple LoRA modules correctly? Thanks in advance.

[jeejeelee]

See: https://docs.vllm.ai/en/latest/models/lora.html#serving-lora-adapters

[andreapairon]

@jeejeelee I saw that, but passing multiple LoRA by using a blank space --lora-modules {name}={path} {name}={path} it's not working in the YAML.

I tried:

lora_modules: "ai-models/adv_perizia_exp7_run6=ai-models/adv_perizia_exp7_run6 ai-models/perizia_exp7_run3=ai-models/perizia_exp7_run3" in which the second LoRA module (ai-models/perizia_exp7_run3=ai-models/perizia_exp7_run3) is separated from the first one by using a blank space.

I receive that stack trace:

Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/ray/serve/_private/application_state.py", line 1049, in build_serve_application
 app = call_app_builder_with_args_if_necessary(import_attr(import_path), args)
File "/usr/local/lib/python3.10/dist-packages/ray/serve/_private/api.py", line 324, in call_app_builder_with_args_if_necessary
 app = builder(args)
File "/serve_app/src/main.py", line 34, in app_builder
 parsed_args = parse_vllm_args(engine_config["engine_kwargs"])
File "/serve_app/src/config/config_parser.py", line 30, in parse_vllm_args
 parsed_args = parser.parse_args(args=arg_strings)
File "/usr/local/lib/python3.10/dist-packages/vllm/utils.py", line 1114, in parse_args
 return super().parse_args(processed_args, namespace)
File "/usr/lib/python3.10/argparse.py", line 1845, in parse_args
 args, argv = self.parse_known_args(args, namespace)

[jeejeelee]

If you have multiple LoRAs, YAML passing is currently not supported. As a temporary solution, you can append args like:

--config  your.yaml --lora_modules lora0=path0 lora1=path1 

[andreapairon]

Ok, thank you for your answer!

[jeejeelee]

BTW, Is passing lora_modules through YAML a common method? If it is commonly used, I can help support this feature.

[andreapairon]

I don't know if it's a common practice, but we currently have a YAML file with engine configs and args that in fact is a Kubernetes config map. Then we mount and read this config map as a Dict and pass them to the arguments parser of vLLM, which, for example, substitute _ with - and add -- at the beginning of the argument name to make the arguments "vLLM-cli compatible".

[jeejeelee]

If that's the case, you can also consider converting the lora_modules in the YAML to a format supported by vLLM. In the YAML, you can write it in list or str type

[andreapairon]

Yeah, I tried as I wrote in my question, but something seems going wrong during the parsing.
I tried as a YAML list:

lora_modules:
      - "ai-models/adv_perizia_exp7_run6=ai-models/adv_perizia_exp7_run6"
      - "ai-models/perizia_exp7_run3=ai-models/perizia_exp7_run3"

[jeejeelee]

If you using --config your.yaml, vllm will convert YAML list to str, see: https://github.com/vllm-project/vllm/blob/main/vllm/utils.py#L1260

[andreapairon]

Uh ok, that is useful. Thanks!

[github-actions]

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!