[ray.llm][Batch] Support LoRA

[comaniac]

Why are these changes needed?

1. Support using LoRA in vLLM v0.
2. Users have to enable LoRA mode in the vLLM engine config, and specify the LoRA adapter in the input request. The adapter will be dynamically loaded to the vLLM engine.
3. The LoRA adapter has to be available in HF hub, or it has to be downloaded to the local file system in advance.

Example usage:

import ray
from ray.data.llm import build_llm_processor, vLLMEngineProcessorConfig

processor_config = vLLMEngineProcessorConfig(
    model="meta-llama/Llama-3.2-1B-Instruct",
    engine_kwargs=dict(
        # Enable LoRA in the vLLM engine.
        enable_lora=True,
        max_lora_rank=32,
    ),
    batch_size=16,
    concurrency=1,
)

processor = build_llm_processor(
    processor_config,
    preprocess=lambda row: dict(
        # Specify LoRA adapter path in the request.
        model="EdBergJr/Llama32_Baha_3",
        messages=[
            {"role": "system", "content": "You are a calculator"},
            {"role": "user", "content": f"{row['id']} ** 3 = ?"},
        ],
        sampling_params=dict(
            temperature=0.3,
            max_tokens=50,
            detokenize=False,
        ),
    ),
    postprocess=lambda row: {
        "resp": row["generated_text"],
    },
)

ds = ray.data.range(60)
ds = ds.map(
    lambda x: {"id": x["id"], "val": x["id"] + 5}
)
ds = processor(ds)
ds = ds.materialize()
for out in ds.take_all():
    print(out)
    print("==========")

cc @gvspraveen @kouroshHakha

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[gvspraveen]

Awesome!!

do you want to update the docs/guides in the sam PR for LoRA?

[kouroshHakha]

add a tiny comment here that says model is lora if and only if the model id present in the row is different than the one in engine_kwargs.

We need to also document this in our docs that for LoRA this is how the preprocessor should be constructed.

[comaniac]

Yes we should document it somewhere later.

[PawaritL]

yes, especially if it's going to be a local path

[kouroshHakha]

big question. Who actually has downloaded the lora adaptor weights? Are those stored in s3? in local? That part I don't understand with the lora support.

[kouroshHakha]

for base model, vllm engine does the downloading which comes from huggingface usually. What about lora here?

[comaniac]

You can download LoRA from HF as well. For example: https://huggingface.co/EdBergJr/Llama32_Baha_3. If the LoRA weights are not available in HF, users have to download it first and specify a local path (in a shared file system). Later we could support loading LoRA weights from S3 directly in vLLM.

[kouroshHakha]

We should document these in the follow up. I think there is so many gotchas here. I know because we have solved all of these issues with the online serving and it's non-trivial. Especially around loading stuff from s3.

In online we have to have a lora-aware router (which we already do through serve multiplex router). Here in ray data, I don't know how that looks like. Maybe we should do a thread safe lora download per node here. or something similar. It needs a bit more thinking (non-blocking to this PR obv)

[comaniac]

@gvspraveen @richardliaw the example is up. Please review and comment: https://anyscale-ray--50804.com.readthedocs.build/en/50804/llm/examples/batch/vllm-with-lora.html

[richardliaw]

Looks great, can you also link the example back into this section: https://anyscale-ray--50804.com.readthedocs.build/en/50804/data/working-with-llms.html#configure-vllm-for-llm-inference

[pcmoritz]

Also vice versa link https://anyscale-ray--50804.com.readthedocs.build/en/50804/data/working-with-llms.html#working-with-llms from the example :)

…

On Mon, Feb 24, 2025 at 1:08 PM Richard Liaw ***@***.***> wrote: Looks great, can you also link the example back into this section: https://anyscale-ray--50804.com.readthedocs.build/en/50804/data/working-with-llms.html#configure-vllm-for-llm-inference — Reply to this email directly, view it on GitHub <#50804 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAA3VJFN6MIXCBQAIZSZPXT2ROC6RAVCNFSM6AAAAABXUAIZNSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMNZZGY2TENJUHE> . You are receiving this because your review was requested.Message ID: ***@***.***> [image: richardliaw]*richardliaw* left a comment (ray-project/ray#50804) <#50804 (comment)> Looks great, can you also link the example back into this section: https://anyscale-ray--50804.com.readthedocs.build/en/50804/data/working-with-llms.html#configure-vllm-for-llm-inference — Reply to this email directly, view it on GitHub <#50804 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAA3VJFN6MIXCBQAIZSZPXT2ROC6RAVCNFSM6AAAAABXUAIZNSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMNZZGY2TENJUHE> . You are receiving this because your review was requested.Message ID: ***@***.***>

[comaniac]

@kouroshHakha @GeneDer could any of you stamp?