How to finetune openvoice with own dataset

[OswaldoBornemann]

How to finetune openvoice with own dataset

[thanhnew2001]

Unfortunately, OpenVoice didn't release the training code and data they use for training. I am working with the finetune script, hope it will work

[thanhnew2001]

I make the script for fine-tuning here: https://github.com/thanhnew2001/OpenVoice_ONNX

[vikassnwloodles]

Hello @thanhnew2001 Kindly help me resolving the following queries:

1. Do we need to use the config.json and checkpoint.pth files from OpenVoice v2’s checkpoint.zip, or should we use different configuration and checkpoint files specifically adapted for the ONNX version?

2. Regarding the dataset structure:

   • What types of audio files should be placed in the training_data folder versus the reference_voices folder?
   • I currently have 2,000 audio files from 5 different speakers. Should I place all 2,000 files in both the training_data and reference_voices directories, or is there a more specific way to partition the data?

Your guidance on these matters would be greatly appreciated to ensure proper setup and alignment with the intended workflow.

Thank you in advance for your time and support.

[DarkKowalski]

Thank you @thanhnew2001, I've tried your fork with both OpenVoice and OpenVoiceV2 checkpoints downloaded from huggingface.

I followed your instruction in README.md and got the following error message repeatedly.

Running on CUDA and my batch size is 1.

Original spec shape: torch.Size([1, 513, 1000])
Reshaped spec: torch.Size([1, 1, 513, 1000])
Error in voice_conversion: Expected 2D (unbatched) or 3D (batched) input to conv1d, but got input of size: [1, 1, 513, 1000]
Spec shape: torch.Size([1, 1, 513, 1000])
src_se shape: torch.Size([1, 256, 1])
ref_se shape: torch.Size([1, 256, 1])
Original spec shape: torch.Size([1, 513, 1000])
Reshaped spec: torch.Size([1, 1, 513, 1000])
Error in voice_conversion: Expected 2D (unbatched) or 3D (batched) input to conv1d, but got input of size: [1, 1, 513, 1000]
Spec shape: torch.Size([1, 1, 513, 1000])
src_se shape: torch.Size([1, 256, 1])
ref_se shape: torch.Size([1, 256, 1])