a simple vanilla example of how to fine tune Llama 2 using Lora adapters for vLLM inference

[raduslmn]

Is anybody kind enough to create a simple vanilla example of how to fine tune Llama 2 using Lora adapters such that it to be later used with vLLM for inference. There is a bit of confusion of whether or not to use quantization when loading the model for fine tuning, apparently vLLM does not work with quantized models. Also which LoraConfig "target_modules" work with vLLM? Also should the adapter be merged?
A simple vanilla example would really help a lot.

[horiacristescu]

Second this

[ai8hyf]

I did something similar for an internal project. Can't share the code but I can talk about the process.

1. Choose a base model and a dataset.
2. Finetune the model using PEFT. 4-bit or 8-bit are optional. (check out the llama2 finetuning script from huggingface/llama repo)
3. The output from step 2 will be an adapter. You need to first merge the adapter with the base model.
4. It's likely that after merging, the model is in safetensor format (not bin). I don't think vLLM supports safetensor, so we need to convert the safetensor model back to bin model. model.save_pretrained and tokenizer.save_pretrained will do the job.
5. You can now load the bin model using vLLM.

[raduslmn]

Thanks @ai8hyf for the reply, i read mixed things about the 4-8 bit with vLLM, some say that a finetuned 4-bit or 8-bit model does not work with vLLM.
Based on what you said, here is the code i came up with. Does this sound like a good approach? I would of course like to be able to train in 4-bit or 8-bit but when i tried loading a 4-8 bit model in vLLM, it did not work.

dataset = load_dataset("DATASET_NAME")
base_model_name = "meta-llama/Llama-2-7b-hf"

dtype = torch.float16

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=dtype,
)

model = AutoModelForCausalLM.from_pretrained(
    base_model_name,
    quantization_config=bnb_config,
    device_map="auto",
    trust_remote_code=True
)
model.config.use_cache = False

model = prepare_model_for_kbit_training(model)

peft_config = LoraConfig(
    lora_alpha=8,
    lora_dropout=0.1,
    r=64,
    bias="none",
    task_type="CAUSAL_LM",
	target_modules = ["q_proj", "v_proj"]
)

tokenizer = AutoTokenizer.from_pretrained(base_model_name, trust_remote_code=True)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "right"

output_dir = "./results"

training_args = TrainingArguments(
    output_dir=output_dir,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    learning_rate=2e-4,
    logging_steps=10,
    max_steps=500,
	optim="paged_adamw_32bit"
)
trainer = SFTTrainer(
    model=model,
    train_dataset=dataset,
    peft_config=peft_config,
    dataset_text_field="text",
    max_seq_length=4096,
    tokenizer=tokenizer,
    args=training_args,
)

trainer.train()
trainer.model.save_pretrained(os.path.join(output_dir, "final_checkpoint"))

#Free memory for merging weights
del model
del trainer
torch.cuda.empty_cache()


# Merge Model with Adapter
model = AutoModelForCausalLM.from_pretrained(
    base_model_name,
    device_map="auto",
    trust_remote_code=True,
    torch_dtype=dtype
)
model.config.use_cache = False
model = PeftModel.from_pretrained(
    model, # The base model with full precision
    os.path.join(output_dir, "final_checkpoint"), # Path to the finetuned adapter
)

model = model.merge_and_unload()
model.save_pretrained(os.path.join(output_dir, "final_merged"), safe_serialization=False)

[ai8hyf]

Thanks @ai8hyf for the reply, i read mixed things about the 4-8 bit with vLLM, some say that a finetuned 4-bit or 8-bit model does not work with vLLM. Based on what you said, here is the code i came up with. Does this sound like a good approach? I would of course like to be able to train in 4-bit or 8-bit but when i tried loading a 4-8 bit model in vLLM, it did not work.

dataset = load_dataset("DATASET_NAME")
base_model_name = "meta-llama/Llama-2-7b-hf"

dtype = torch.float16

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=dtype,
)

model = AutoModelForCausalLM.from_pretrained(
    base_model_name,
    quantization_config=bnb_config,
    device_map="auto",
    trust_remote_code=True
)
model.config.use_cache = False

model = prepare_model_for_kbit_training(model)

peft_config = LoraConfig(
    lora_alpha=8,
    lora_dropout=0.1,
    r=64,
    bias="none",
    task_type="CAUSAL_LM",
	target_modules = ["q_proj", "v_proj"]
)

tokenizer = AutoTokenizer.from_pretrained(base_model_name, trust_remote_code=True)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "right"

output_dir = "./results"

training_args = TrainingArguments(
    output_dir=output_dir,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    learning_rate=2e-4,
    logging_steps=10,
    max_steps=500,
	optim="paged_adamw_32bit"
)
trainer = SFTTrainer(
    model=model,
    train_dataset=dataset,
    peft_config=peft_config,
    dataset_text_field="text",
    max_seq_length=4096,
    tokenizer=tokenizer,
    args=training_args,
)

trainer.train()
trainer.model.save_pretrained(os.path.join(output_dir, "final_checkpoint"))

#Free memory for merging weights
del model
del trainer
torch.cuda.empty_cache()


# Merge Model with Adapter
model = AutoModelForCausalLM.from_pretrained(
    base_model_name,
    device_map="auto",
    trust_remote_code=True,
    torch_dtype=dtype
)
model.config.use_cache = False
model = PeftModel.from_pretrained(
    model, # The base model with full precision
    os.path.join(output_dir, "final_checkpoint"), # Path to the finetuned adapter
)

model = model.merge_and_unload()
model.save_pretrained(os.path.join(output_dir, "final_merged"), safe_serialization=False)

Sorry if I didn't made it clear. I don't think vLLM supports quantized models at this point. However, you can still end up with a fp16 model after finetuning with QLoRA. The fine-tuning process with QLoRA is either in 4bit or 8bit but the final LoRA adapter works with fp16 models. The merged models are also in fp16.

[sandys]

folks any update on this ? we are looking to run finetuned models as well

[BRM10213]

It is easy to do it.

First, you have to config an environment with autotrain:

pip3 install autotrain-advanced
pip3 install torchvision
pip3 install peft

Make your custom train with this command (tested by A100 gpu):

nohup autotrain llm \
--train \
--text_column 'text'  \
--project-name 'CodeLlama-34b-Instruct-001'  \
--model 'codellama/CodeLlama-34b-Instruct-hf'  \
--data-path '/home/azureuser/Fine_Tuning'  \
--peft  \
--quantization int8  \
--lr 2e-4  \
--batch-size 10  \
--epochs 8  \
--trainer sft &

Use this python script to do a merge from your base model with all changes from your training model:

import torch

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

def merge_adapter(base_model_path, target_model_path, adapter_path):
    """
    Fusiona un adaptador entrenado con un modelo base para crear un modelo completo y guarda el modelo fusionado.

    Args:
        base_model_path (str): Ruta del modelo base.
        target_model_path (str): Ruta donde se guardará el modelo fusionado.
        adapter_path (str): Ruta del adaptador entrenado.

    Returns:
        None
    """
    # Cargar el modelo base
    print("Loading adapter...")
    model = AutoModelForCausalLM.from_pretrained(
        base_model_path,
        torch_dtype=torch.float16,  # Establecer el tipo de dato de PyTorch para la carga del modelo
        low_cpu_mem_usage=True,     # Habilitar el uso reducido de memoria de CPU
        trust_remote_code=True,     # Confiar en el código remoto del modelo
    )

    # Cargar el tokenizador del modelo de destino
    tokenizer = AutoTokenizer.from_pretrained(
        adapter_path,
        trust_remote_code=True,  # Confiar en el código remoto del tokenizador
    )
    # Ajustar el tamaño de los embeddings de los tokens para que coincida con el tamaño del vocabulario del tokenizador
    model.resize_token_embeddings(len(tokenizer))

    # Cargar el adaptador y fusionarlo con el modelo base
    model = PeftModel.from_pretrained(model, adapter_path)
    model = model.merge_and_unload()  # Fusionar el adaptador con el modelo base y liberar el adaptador

    # Guardar el modelo fusionado y el tokenizador en la ruta especificada
    print("Saving target model...")
    model.save_pretrained(target_model_path)
    tokenizer.save_pretrained(target_model_path)


if __name__ == "__main__":
    base_model_path = "codellama/CodeLlama-34b-Instruct-hf"
    target_model_path = "/home/azureuser/CodeLlama-34b-Instruct-001-Fused"
    adapter_path = "/home/azureuser/CodeLlama-34b-Instruct-001"

    merge_adapter(base_model_path, target_model_path, adapter_path)

Lastly, optional, you can use fastchat with vllm to serve it.

python3 -m fastchat.serve.vllm_worker --model-path /home/azureuser/CodeLlama-34b-Instruct-001-Fused --controller-address http://localhost:21001 --port 21002 --worker-address http://localhost:21002 --num-gpus 1

I hope this can be usefull for someone.

[hmellor]

Closing this because a fine-tuning tutorial is out of scope for vLLM.