Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: Naive finetuning #93

Merged
merged 2 commits into from
Mar 13, 2024
Merged

feat: Naive finetuning #93

merged 2 commits into from
Mar 13, 2024

Conversation

lucapericlp
Copy link
Contributor

Taking the approach of freezing the entire LLM other than the [-last_n_blocks_to_finetune:] & the last norm layer. Using the "default" hyperparams, we're seeing ~12Gb VRAM usage.

lucapericlp
lucapericlp commented Mar 13, 2024
View reviewed changes
datasets/sample_dataset.csv
@@ -0,0 +1,321 @@
audio_files|captions
Copy link
Contributor Author

@lucapericlp lucapericlp Mar 13, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Opted to omit from git lfs to avoid setup complexity (since we don't really gain much)

vatsalaggarwal
vatsalaggarwal previously approved these changes Mar 13, 2024
View reviewed changes
Copy link
Member

@vatsalaggarwal vatsalaggarwal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice!

fam/llm/config/finetune_params.py Outdated Show resolved Hide resolved
fam/llm/preprocessing/audio_token_mode.py Outdated Show resolved Hide resolved
README.md Outdated Show resolved Hide resolved
@vatsalaggarwal vatsalaggarwal mentioned this pull request Mar 13, 2024
Draft
@lucapericlp lucapericlp force-pushed the task/poetry branch 2 times, most recently from 358fc04 to d18144c Compare March 13, 2024 15:29
@lucapericlp lucapericlp force-pushed the task/naive-finetuning-unoptimised branch from 84db8e9 to 5efe403 Compare March 13, 2024 15:40
@lucapericlp lucapericlp changed the base branch from task/poetry to main March 13, 2024 15:42
@lucapericlp lucapericlp dismissed vatsalaggarwal’s stale review March 13, 2024 15:42

The base branch was changed.

@lucapericlp lucapericlp force-pushed the task/naive-finetuning-unoptimised branch from 5efe403 to a5bf07d Compare March 13, 2024 15:51
vatsalaggarwal
vatsalaggarwal previously approved these changes Mar 13, 2024
View reviewed changes
Copy link
Member

@vatsalaggarwal vatsalaggarwal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

small nit, good to go!

fam/llm/config/finetune_params.py Outdated Show resolved Hide resolved
@lucapericlp lucapericlp force-pushed the task/naive-finetuning-unoptimised branch from 8ddbfe3 to c376ead Compare March 13, 2024 16:21
@vatsalaggarwal vatsalaggarwal merged commit 62477e7 into main Mar 13, 2024
@vatsalaggarwal vatsalaggarwal deleted the task/naive-finetuning-unoptimised branch March 13, 2024 16:21
@sidroopdaska sidroopdaska mentioned this pull request Mar 14, 2024
Open
This was referenced Mar 14, 2024
Open
Open
Closed
Open
vatsalaggarwal added a commit that referenced this pull request Mar 14, 2024
@vatsalaggarwal
Merge branch 'main' into vatsal/quantization
508cb0d 
* main:
  feat: naive finetuning (#93)
  Adding dependency versioning via poetry (#92)
@platform-kit platform-kit mentioned this pull request Mar 14, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@vatsalaggarwal vatsalaggarwal vatsalaggarwal approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@lucapericlp @vatsalaggarwal