Fine-tuning DPO Script

[arhaankk]

• contains fine-tuning script with DPO and LoRA and PEFT + 4-bit Quantization.

Request for full dataset with DPO format @jie-jw-wu

reference:

• http://github.com/mlabonne/llm-course/blob/main/Fine_tune_a_Mistral_7b_model_with_DPO.ipynb
• https://github.com/raghavc/LLM-RLHF-Tuning-with-PPO-and-DPO/blob/main/script/dpo/run_dpo_with_peft.py

[jie-jw-wu]

check what are the difference between these 2 fields

[arhaankk]

max_prompt_length controls only the prompt (input). Defines the maximum allowed length of the prompt (input sequence) in tokens.
max_length controls the total length (input + output). Defines the maximum allowed length for the entire sequence.