Tips for training a custom voice?

[Clayton-Klemm]

I'll take any tips I can get. I've tried longer recording samples, I've tried shorter samples. It still just is not good. I've tried anything ranging from 20 seconds to close to 3 minutes. Is it better to go really long? like 10 minutes? half an hour? I found the sample usage in the demo3 notebook to be a bit misleading due to directory differences and a few other differences. This is how I'm currently using the repo

import nltk
nltk.download('averaged_perceptron_tagger')
nltk.download('punkt')

import os
import torch
from openvoice import se_extractor
from openvoice.api import ToneColorConverter
from melo.api import TTS

class VoiceService:
    def __init__(self):
        self._ckpt_converter = 'checkpoints_v2/converter'
        self._device = "cuda:0" if torch.cuda.is_available() else "cpu"
        self._output_dir = 'outputs_v2'
        os.makedirs(self._output_dir, exist_ok=True)

        # Load the tone color converter model
        self._tone_color_converter = ToneColorConverter(
            f'{self._ckpt_converter}/config.json', device=self._device)
        self._tone_color_converter.load_ckpt(f'{self._ckpt_converter}/checkpoint.pth')
    
    def clone_voice(self, reference_speaker, text, output_filename, language='EN', base_speaker_key='EN-US'):
        # Extract the speaker embedding (tone color) from your voice sample
        target_se, _ = se_extractor.get_se(
            reference_speaker, self._tone_color_converter, vad=True)

        # Initialize the TTS model for the specified language
        model = TTS(language=language, device=self._device)
        
        # Get the speaker ID for the base speaker
        speaker_ids = model.hps.data.spk2id
        print("Available base speakers:", list(speaker_ids.keys()))
        if base_speaker_key not in speaker_ids:
            raise ValueError(f"Base speaker key '{base_speaker_key}' not found. Available keys: {list(speaker_ids.keys())}")
        speaker_id = speaker_ids[base_speaker_key]

        # Load the source speaker embedding for the base speaker
        source_se_path = f'checkpoints_v2/base_speakers/ses/{base_speaker_key}.pth'
        if not os.path.exists(source_se_path):
            raise FileNotFoundError(f"Source speaker embedding not found at '{source_se_path}'")
        source_se = torch.load(source_se_path, map_location=self._device)
        
        # Generate TTS audio using the base speaker
        tmp_src_path = os.path.join(self._output_dir, 'tmp.wav')
        speed = .9  # Adjust the speed if necessary
        model.tts_to_file(text, speaker_id, tmp_src_path, speed=speed)
        
        # Convert the TTS audio to your voice using the tone color converter
        output_path = os.path.join(self._output_dir, output_filename)
        self._tone_color_converter.convert(
            audio_src_path=tmp_src_path, 
            src_se=source_se, 
            tgt_se=target_se, 
            output_path=output_path,
            message="@MyShell")
        
        print(f"Generated audio saved at '{output_path}'")

if __name__ == "__main__":
    service = VoiceService()
    reference_speaker = 'resources/output.wav'  # Path to your voice recording
    text = """
        Large Language Models (LLMs) have recently demonstrated remarkable capabilities in natural language processing tasks and
        beyond. This success of LLMs has led to a large influx of research contributions in this direction. These works encompass diverse
        topics such as architectural innovations, better training strategies, context length improvements, fine-tuning, multi-modal LLMs,
        robotics, datasets, benchmarking, efficiency, and more. With the rapid development of techniques and regular breakthroughs in
        LLM research, it has become considerably challenging to perceive the bigger picture of the advances in this direction. Considering
        the rapidly emerging plethora of literature on LLMs, it is imperative that the research community is able to benefit from a concise
        yet comprehensive overview of the recent developments in this field.
    """

    output_filename = 'my_cloned_voice_output5.wav'
    language = 'EN'  # Language code should be 'EN'
    base_speaker_key = 'EN-Default'  # Use one of the available base speaker keys ['EN-US', 'EN-BR', 'EN_INDIA', 'EN-AU', 'EN-Default']

    service.clone_voice(reference_speaker, text, output_filename, language, base_speaker_key)

[dosubot[bot]]

@Clayton-Klemm Hello there! I'm here to help you with any bugs, questions, or contributions you have.

To improve the quality of your custom voice training using OpenVoice, here are some tips:

1. Ensure Clean Reference Audio: The reference audio should be clean and free from background noise. High-quality reference speech can be found here.

2. Use Longer Recording Samples: Avoid short audio clips as they may lead to poor quality. Using longer recording samples, such as 10 minutes or even half an hour, is advisable for better results ^[1].

3. Single Speaker Audio: Ensure that the reference audio contains speech from only one person.

4. Avoid Long Blank Sections: The reference audio should not contain long blank sections.

5. Correct Naming: Ensure that you do not reuse the same name for the reference audio without deleting the processed folder.

Additionally, make sure to adjust the speed parameter in the TTS model to find the most natural-sounding speech for your specific use case. Typical values range around 1.0 for normal speed, but you can experiment with values like 0.9 for a slightly slower and more deliberate speech style ^[2][3].

By following these tips, you should be able to achieve better results in cloning a custom voice.

To continue talking to Dosu, mention @dosu.

[MeachamusPrime]

I have followed all of @dosu's suggestions (which are effectively a rehash of the QnA.md) and have seen no improvement. The output sounds nothing like the reference speaker. It sounds more like some guy making a poor attempt at faking the voice than a cloning of the actual voice. @Clayton-Klemm, did you ever find a way to get it closer to a clone of the voice?