Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Tiny] renaming a field name (init_lora_weights) #1236

Merged
merged 3 commits into from
Jan 26, 2025
Merged

[Tiny] renaming a field name (init_lora_weights) #1236

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
fcafdc7
update
optas Jan 25, 2025
ed82dd6
update
optas Jan 25, 2025
8307af4
Merge branch 'main' into optas/tiny_change_field_name
optas Jan 25, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion configs/recipes/vision/llama3_2_vision/sft/11b_lora/train.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ peft:
- "gate_proj"
- "up_proj"
- "down_proj"
init_lora_weights: DEFAULT
lora_init_weights: DEFAULT

fsdp:
enable_fsdp: True
Expand Down
2 changes: 1 addition & 1 deletion docs/user_guides/train/configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -228,7 +228,7 @@ peft:
lora_modules_to_save: null # Modules to unfreeze and train
lora_bias: "none" # Bias training type
lora_task_type: "CAUSAL_LM" # Task type for adaptation
init_lora_weights: "DEFAULT" # Initialization of LoRA weights
lora_init_weights: "DEFAULT" # Initialization of LoRA weights

# Q-LoRA settings
q_lora: false # Enable quantization
Expand Down
4 changes: 2 additions & 2 deletions src/oumi/builders/models.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -448,7 +448,7 @@ def build_tokenizer(
return tokenizer


def _convert_init_lora_weights_to_lora_config(
def _convert_lora_init_weights_to_lora_config(
param: LoraWeightInitialization,
) -> Union[
bool,
Expand Down Expand Up @@ -491,7 +491,7 @@ def build_peft_model(
bias=peft_params.lora_bias, # type: ignore
task_type=peft_params.lora_task_type,
init_lora_weights=(
_convert_init_lora_weights_to_lora_config(peft_params.init_lora_weights)
_convert_lora_init_weights_to_lora_config(peft_params.lora_init_weights)
),
)

Expand Down
2 changes: 1 addition & 1 deletion src/oumi/core/configs/params/peft_params.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -161,7 +161,7 @@ class PeftParams(BaseParams):
https://github.com/huggingface/peft/blob/main/src/peft/tuners/lora/config.py
"""

init_lora_weights: LoraWeightInitialization = field(
lora_init_weights: LoraWeightInitialization = field(
default=LoraWeightInitialization.DEFAULT,
metadata={
"help": "Weights initialization for LoRA adapters.",
Expand Down