[Core] Relax the LoRA max rank #26461
Conversation
jeejeelee
requested review from
ProExpertProg,
WoosukKwon,
alexm-redhat,
comaniac,
hmellor,
houseroad,
mgoin,
njhill,
robertgshaw2-redhat,
simon-mo,
tlrmchlsmth,
yewentao256,
youkaichao and
ywang96
as code owners
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request aims to add support for max_lora_rank=1. The changes in vllm/config/lora.py and vllm/v1/worker/lora_model_runner_mixin.py are correct for this purpose. However, I've identified a significant performance issue related to CUDA graph capture for LoRA, which is detailed in my comment.
| lora_warmup_rank = ( | ||
| lora_config.max_lora_rank if lora_config.max_lora_rank < 8 else 8 | ||
| ) |
There was a problem hiding this comment.
This logic for determining lora_warmup_rank is correct for supporting max_lora_rank=1, but it perpetuates a significant performance issue for CUDA graph capture when max_lora_rank > 8.
The _dummy_run method, which uses this lora_warmup_rank, is responsible for both profiling and CUDA graph capture. By capping the warmup rank at 8, any captured CUDA graph will be specific to LoRA ranks up to 8. If the engine is configured with max_lora_rank > 8 (e.g., 16) and receives requests with LoRAs of a rank greater than 8, the captured graph will not be used, causing a fallback to eager execution and negating the performance benefits of CUDA graphs.
To ensure CUDA graphs are captured for the maximum configured rank, the warmup rank should be lora_config.max_lora_rank during graph capture. A simple fix is to always use max_lora_rank, which would also make it consistent with the TPU runner. This might slow down warm-up for large ranks, but it ensures that CUDA graphs are effective for all configured LoRA ranks.
lora_warmup_rank = lora_config.max_lora_rank
github-actions
bot
added
the
ready
…to loader * 'loader' of https://github.com/dsxsteven/vllm_splitPR: (778 commits) [torchao] Add support for ModuleFqnToConfig using regex (vllm-project#26001) Add: Support for multiple hidden layers in Eagle3 (vllm-project#26164) Enable `RMSNorm` substitution for Transformers backend (vllm-project#26353) [Model] Gemma3: Fix GGUF loading and quantization (vllm-project#26189) Bump Flashinfer to v0.4.0 (vllm-project#26326) Update Dockerfile and install runai-model-streamer[gcs] package (vllm-project#26464) [Core] Relax the LoRA max rank (vllm-project#26461) [CI/Build] Fix model nightly tests (vllm-project#26466) [Hybrid]: Decouple Kernel Block Size from KV Page Size (vllm-project#24486) [Core][KVConnector] Propagate all tokens on resumed preemptions (vllm-project#24926) [MM][Doc] Add documentation for configurable mm profiling (vllm-project#26200) [Hardware][AMD] Enable FlexAttention backend on ROCm (vllm-project#26439) [Bugfix] Incorrect another MM data format in vllm bench throughput (vllm-project#26462) [Bugfix] Catch and log invalid token ids in detokenizer #2 (vllm-project#26445) [Minor] Change warning->warning_once in preprocess (vllm-project#26455) [Bugfix] Set the minimum python version for gpt-oss (vllm-project#26392) [Misc] Redact ray runtime env before logging (vllm-project#26302) Separate MLAAttention class from Attention (vllm-project#25103) [Attention] Register FLASHMLA_SPARSE (vllm-project#26441) [Kernels] Modular kernel refactor (vllm-project#24812) ...
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com> Signed-off-by: xuebwang-amd <xuebwang@amd.com>
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com> Signed-off-by: Dhruvil Bhatt <bhattdbh@amazon.com>
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com> Signed-off-by: xuebwang-amd <xuebwang@amd.com>
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com> Signed-off-by: 0xrushi <6279035+0xrushi@users.noreply.github.com>
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com> Signed-off-by: 0xrushi <6279035+0xrushi@users.noreply.github.com>
3 participants
Purpose
FIX #26428
Although we have already supported lora rank=1, we are adding a max lora rank=1 option to accommodate scenarios where only LoRA rank=1 is needed.
also cc @varun-sundar-rabindranath
Test Plan
Test Result
Essential Elements of an Effective PR Description Checklist
supported_models.mdandexamplesfor a new model.