Skip to content

Commit

Permalink
feat: naive finetuning (metavoiceio#93)
Browse files Browse the repository at this point in the history 
* Adding variable transformer block finetuning with documentation & some defaults

* Moving last bit over to finetuning file
  • Loading branch information
@lucapericlp
lucapericlp authored Mar 13, 2024
1 parent b236bcf commit 62477e7
Show file tree
Hide file tree
Showing 19 changed files with 1,816 additions and 12 deletions.
30 changes: 29 additions & 1 deletion README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -78,10 +78,38 @@ poetry run python app.py
3. Use it via [Hugging Face](https://huggingface.co/metavoiceio)
4. [Google Collab Demo](https://colab.research.google.com/github/metavoiceio/metavoice-src/blob/main/colab_demo.ipynb)

## Finetuning
We support finetuning the first stage LLM (see [Architecture section](#Architecture)).

In order to finetune, we expect a "|"-delimited CSV dataset of the following format:

```csv
audio_files|captions
./data/audio.wav|./data/caption.txt
```

Note that we don't perform any dataset overlap checks, so ensure that your train and val datasets are disjoint.

Try it out using our sample datasets via:
```bash
poetry run finetune --train ./datasets/sample_dataset.csv --val ./datasets/sample_val_dataset.csv
```

### Configuration

In order to set hyperparameters such as learning rate, what to freeze, etc, you
can edit the [finetune_params.py](./fam/llm/config/finetune_params.py) file.

We've got a light & optional integration with W&B that can be enabled via setting
`wandb_log = True` & by installing the appropriate dependencies.

```bash
poetry install -E observable
```

## Upcoming
- [x] Faster inference ⚡
- [ ] Fine-tuning code
- [x] Fine-tuning code 📐
- [ ] Synthesis of arbitrary length text


Expand Down
1 change: 1 addition & 0 deletions data/caption.txt
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Please call Stella.
321 changes: 321 additions & 0 deletions datasets/sample_dataset.csv
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,321 @@
audio_files|captions
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
./data/audio.wav|./data/caption.txt
Loading

0 comments on commit 62477e7

Please sign in to comment.