Major maintenance update (#961)

README.md

@@ -1,8 +1,8 @@
 # Real-Time Voice Cloning
 This repository is an implementation of [Transfer Learning from Speaker Verification to
-Multispeaker Text-To-Speech Synthesis](https://arxiv.org/pdf/1806.04558.pdf) (SV2TTS) with a vocoder that works in real-time. Feel free to check [my thesis](https://matheo.uliege.be/handle/2268.2/6801) if you're curious or if you're looking for info I haven't documented. Mostly I would recommend giving a quick look to the figures beyond the introduction.
+Multispeaker Text-To-Speech Synthesis](https://arxiv.org/pdf/1806.04558.pdf) (SV2TTS) with a vocoder that works in real-time. This was my [master's thesis](https://matheo.uliege.be/handle/2268.2/6801).
 
-SV2TTS is a three-stage deep learning framework that allows to create a numerical representation of a voice from a few seconds of audio, and to use it to condition a text-to-speech model trained to generalize to new voices.
+SV2TTS is a deep learning framework in three stages. In the first stage, one creates a digital representation of a voice from a few seconds of audio. In the second and third stages, this representation is used as reference to generate speech given arbitrary text.
 
 **Video demonstration** (click the picture):
 
@@ -19,31 +19,28 @@ SV2TTS is a three-stage deep learning framework that allows to create a numerica
 |[1710.10467](https://arxiv.org/pdf/1710.10467.pdf) | GE2E (encoder)| Generalized End-To-End Loss for Speaker Verification | This repo |
 
 ## News
+**28/12/21**: I've done a [major maintenance update](https://github.com/CorentinJ/Real-Time-Voice-Cloning/pull/961). Mostly, I've worked on making setup easier. Find new instructions in the section below.
+
 **14/02/21**: This repo now runs on PyTorch instead of Tensorflow, thanks to the help of @bluefish. If you wish to run the tensorflow version instead, checkout commit `5425557`.
 
 **13/11/19**: I'm now working full time and I will not maintain this repo anymore. To anyone who reads this:
 - **If you just want to clone your voice (and not someone else's):** I recommend our free plan on [Resemble.AI](https://www.resemble.ai/). You will get a better voice quality and less prosody errors.
 - **If this is not your case:** proceed with this repository, but you might end up being disappointed by the results. If you're planning to work on a serious project, my strong advice: find another TTS repo. Go [here](https://github.com/CorentinJ/Real-Time-Voice-Cloning/issues/364) for more info.
 
-**20/08/19:** I'm working on [resemblyzer](https://github.com/resemble-ai/Resemblyzer), an independent package for the voice encoder. You can use your trained encoder models from this repo with it.
-
-**06/07/19:** Need to run within a docker container on a remote server? See [here](https://sean.lane.sh/posts/2019/07/Running-the-Real-Time-Voice-Cloning-project-in-Docker/).
-
-**25/06/19:** Experimental support for low-memory GPUs (~2gb) added for the synthesizer. Pass `--low_mem` to `demo_cli.py` or `demo_toolbox.py` to enable it. It adds a big overhead, so it's not recommended if you have enough VRAM.
+**20/08/19:** I'm working on [resemblyzer](https://github.com/resemble-ai/Resemblyzer), an independent package for the voice encoder (inference only). You can use your trained encoder models from this repo with it.
 
 
 ## Setup
 
-### 1. Install Requirements
-
-**Python 3.6 or 3.7** is needed to run the toolbox.
-
-* Install [PyTorch](https://pytorch.org/get-started/locally/) (>=1.1.0).
-* Install [ffmpeg](https://ffmpeg.org/download.html#get-packages).
-* Run `pip install -r requirements.txt` to install the remaining necessary packages.
+#### 1. Install Requirements
+1. Both Windows and Linux and supported. A GPU is recommended for training and for inference speed, but is not mandatory.
+2. Python 3.7 or greater is recommended. Python 3.6 should work, python 3.5 might need some tweaking. I recommend setting up a virtual environment using `venv`, but this is optional.
+3. Install [ffmpeg](https://ffmpeg.org/download.html#get-packages). This is necessary for reading audio files.
+4. Install [PyTorch](https://pytorch.org/get-started/locally/). Pick the latest stable version, your operating system, your package manager (pip by default) and finally pick any of the proposed CUDA versions if you have a GPU, otherwise pick CPU. Run the given command.
+5. Install the remaining requirements with `pip install -r requirements.txt`
 
-### 2. Download Pretrained Models
-Download the latest [here](https://github.com/CorentinJ/Real-Time-Voice-Cloning/wiki/Pretrained-models).
+### 2. (Optional) Download Pretrained Models
+Pretrained models are now downloaded automatically. If this doesn't work for you, you can manually download them [here](https://github.com/CorentinJ/Real-Time-Voice-Cloning/wiki/Pretrained-models).
 
 ### 3. (Optional) Test Configuration
 Before you download any dataset, you can begin by testing your configuration with:

demo_cli.py

@@ -1,66 +1,56 @@
-from encoder.params_model import model_embedding_size as speaker_embedding_size
-from utils.argutils import print_args
-from utils.modelutils import check_model_paths
-from synthesizer.inference import Synthesizer
-from encoder import inference as encoder
-from vocoder import inference as vocoder
+import argparse
+import os
 from pathlib import Path
+
+import librosa
 import numpy as np
 import soundfile as sf
-import librosa
-import argparse
 import torch
-import sys
-import os
 from audioread.exceptions import NoBackendError
 
+from encoder import inference as encoder
+from encoder.params_model import model_embedding_size as speaker_embedding_size
+from synthesizer.inference import Synthesizer
+from utils.argutils import print_args
+from utils.default_models import ensure_default_models
+from vocoder import inference as vocoder
+
+
 if __name__ == '__main__':
-    ## Info & args
     parser = argparse.ArgumentParser(
         formatter_class=argparse.ArgumentDefaultsHelpFormatter
     )
-    parser.add_argument("-e", "--enc_model_fpath", type=Path, 
-                        default="encoder/saved_models/pretrained.pt",
+    parser.add_argument("-e", "--enc_model_fpath", type=Path,
+                        default="saved_models/default/encoder.pt",
                         help="Path to a saved encoder")
-    parser.add_argument("-s", "--syn_model_fpath", type=Path, 
-                        default="synthesizer/saved_models/pretrained/pretrained.pt",
+    parser.add_argument("-s", "--syn_model_fpath", type=Path,
+                        default="saved_models/default/synthesizer.pt",
                         help="Path to a saved synthesizer")
-    parser.add_argument("-v", "--voc_model_fpath", type=Path, 
-                        default="vocoder/saved_models/pretrained/pretrained.pt",
+    parser.add_argument("-v", "--voc_model_fpath", type=Path,
+                        default="saved_models/default/vocoder.pt",
                         help="Path to a saved vocoder")
     parser.add_argument("--cpu", action="store_true", help=\
         "If True, processing is done on CPU, even when a GPU is available.")
     parser.add_argument("--no_sound", action="store_true", help=\
         "If True, audio won't be played.")
     parser.add_argument("--seed", type=int, default=None, help=\
         "Optional random number seed value to make toolbox deterministic.")
-    parser.add_argument("--no_mp3_support", action="store_true", help=\
-        "If True, disallows loading mp3 files to prevent audioread errors when ffmpeg is not installed.")
     args = parser.parse_args()
+    arg_dict = vars(args)
     print_args(args, parser)
-    if not args.no_sound:
-        import sounddevice as sd
 
-    if args.cpu:
-        # Hide GPUs from Pytorch to force CPU processing
+    # Hide GPUs from Pytorch to force CPU processing
+    if arg_dict.pop("cpu"):
         os.environ["CUDA_VISIBLE_DEVICES"] = "-1"
 
-    if not args.no_mp3_support:
-        try:
-            librosa.load("samples/1320_00000.mp3")
-        except NoBackendError:
-            print("Librosa will be unable to open mp3 files if additional software is not installed.\n"
-                  "Please install ffmpeg or add the '--no_mp3_support' option to proceed without support for mp3 files.")
-            exit(-1)
-
     print("Running a test of your configuration...\n")
-        
+
     if torch.cuda.is_available():
         device_id = torch.cuda.current_device()
         gpu_properties = torch.cuda.get_device_properties(device_id)
         ## Print some environment information (for debugging purposes)
         print("Found %d GPUs available. Using GPU %d (%s) of compute capability %d.%d with "
-            "%.1fGb total memory.\n" % 
+            "%.1fGb total memory.\n" %
             (torch.cuda.device_count(),
             device_id,
             gpu_properties.name,
@@ -69,68 +59,64 @@
             gpu_properties.total_memory / 1e9))
     else:
         print("Using CPU for inference.\n")
-
-    ## Remind the user to download pretrained models if needed
-    check_model_paths(encoder_path=args.enc_model_fpath,
-                      synthesizer_path=args.syn_model_fpath,
-                      vocoder_path=args.voc_model_fpath)
-
+
     ## Load the models one by one.
     print("Preparing the encoder, the synthesizer and the vocoder...")
+    ensure_default_models(Path("saved_models"))
     encoder.load_model(args.enc_model_fpath)
     synthesizer = Synthesizer(args.syn_model_fpath)
     vocoder.load_model(args.voc_model_fpath)
-    
-    
+
+
     ## Run a test
     print("Testing your configuration with small inputs.")
     # Forward an audio waveform of zeroes that lasts 1 second. Notice how we can get the encoder's
     # sampling rate, which may differ.
-    # If you're unfamiliar with digital audio, know that it is encoded as an array of floats 
+    # If you're unfamiliar with digital audio, know that it is encoded as an array of floats
     # (or sometimes integers, but mostly floats in this projects) ranging from -1 to 1.
     # The sampling rate is the number of values (samples) recorded per second, it is set to
-    # 16000 for the encoder. Creating an array of length <sampling_rate> will always correspond 
+    # 16000 for the encoder. Creating an array of length <sampling_rate> will always correspond
     # to an audio of 1 second.
     print("\tTesting the encoder...")
     encoder.embed_utterance(np.zeros(encoder.sampling_rate))
-    
+
     # Create a dummy embedding. You would normally use the embedding that encoder.embed_utterance
     # returns, but here we're going to make one ourselves just for the sake of showing that it's
     # possible.
     embed = np.random.rand(speaker_embedding_size)
-    # Embeddings are L2-normalized (this isn't important here, but if you want to make your own 
+    # Embeddings are L2-normalized (this isn't important here, but if you want to make your own
     # embeddings it will be).
     embed /= np.linalg.norm(embed)
-    # The synthesizer can handle multiple inputs with batching. Let's create another embedding to 
+    # The synthesizer can handle multiple inputs with batching. Let's create another embedding to
     # illustrate that
     embeds = [embed, np.zeros(speaker_embedding_size)]
     texts = ["test 1", "test 2"]
     print("\tTesting the synthesizer... (loading the model will output a lot of text)")
     mels = synthesizer.synthesize_spectrograms(texts, embeds)
-    
-    # The vocoder synthesizes one waveform at a time, but it's more efficient for long ones. We 
+
+    # The vocoder synthesizes one waveform at a time, but it's more efficient for long ones. We
     # can concatenate the mel spectrograms to a single one.
     mel = np.concatenate(mels, axis=1)
-    # The vocoder can take a callback function to display the generation. More on that later. For 
+    # The vocoder can take a callback function to display the generation. More on that later. For
     # now we'll simply hide it like this:
     no_action = lambda *args: None
     print("\tTesting the vocoder...")
-    # For the sake of making this test short, we'll pass a short target length. The target length 
-    # is the length of the wav segments that are processed in parallel. E.g. for audio sampled 
+    # For the sake of making this test short, we'll pass a short target length. The target length
+    # is the length of the wav segments that are processed in parallel. E.g. for audio sampled
     # at 16000 Hertz, a target length of 8000 means that the target audio will be cut in chunks of
-    # 0.5 seconds which will all be generated together. The parameters here are absurdly short, and 
-    # that has a detrimental effect on the quality of the audio. The default parameters are 
+    # 0.5 seconds which will all be generated together. The parameters here are absurdly short, and
+    # that has a detrimental effect on the quality of the audio. The default parameters are
     # recommended in general.
     vocoder.infer_waveform(mel, target=200, overlap=50, progress_callback=no_action)
-    
+
     print("All test passed! You can now synthesize speech.\n\n")
-    
-    
+
+
     ## Interactive speech generation
     print("This is a GUI-less example of interface to SV2TTS. The purpose of this script is to "
           "show how you can interface this project easily with your own. See the source code for "
           "an explanation of what is happening.\n")
-    
+
     print("Interactive generation loop")
     num_generated = 0
     while True:
@@ -140,31 +126,28 @@
                       "wav, m4a, flac, ...):\n"
             in_fpath = Path(input(message).replace("\"", "").replace("\'", ""))
 
-            if in_fpath.suffix.lower() == ".mp3" and args.no_mp3_support:
-                print("Can't Use mp3 files please try again:")
-                continue
             ## Computing the embedding
-            # First, we load the wav using the function that the speaker encoder provides. This is 
+            # First, we load the wav using the function that the speaker encoder provides. This is
             # important: there is preprocessing that must be applied.
-            
+
             # The following two methods are equivalent:
             # - Directly load from the filepath:
             preprocessed_wav = encoder.preprocess_wav(in_fpath)
             # - If the wav is already loaded:
             original_wav, sampling_rate = librosa.load(str(in_fpath))
             preprocessed_wav = encoder.preprocess_wav(original_wav, sampling_rate)
             print("Loaded file succesfully")
-            
-            # Then we derive the embedding. There are many functions and parameters that the 
+
+            # Then we derive the embedding. There are many functions and parameters that the
             # speaker encoder interfaces. These are mostly for in-depth research. You will typically
             # only use this function (with its default parameters):
             embed = encoder.embed_utterance(preprocessed_wav)
             print("Created the embedding")
-            
-            
+
+
             ## Generating the spectrogram
             text = input("Write a sentence (+-20 words) to be synthesized:\n")
-            
+
             # If seed is specified, reset torch seed and force synthesizer reload
             if args.seed is not None:
                 torch.manual_seed(args.seed)
@@ -178,8 +161,8 @@
             specs = synthesizer.synthesize_spectrograms(texts, embeds)
             spec = specs[0]
             print("Created the mel spectrogram")
-            
-            
+
+
             ## Generating the waveform
             print("Synthesizing the waveform:")
 
@@ -191,18 +174,19 @@
             # Synthesizing the waveform is fairly straightforward. Remember that the longer the
             # spectrogram, the more time-efficient the vocoder.
             generated_wav = vocoder.infer_waveform(spec)
-            
-            
+
+
             ## Post-generation
             # There's a bug with sounddevice that makes the audio cut one second earlier, so we
             # pad it.
             generated_wav = np.pad(generated_wav, (0, synthesizer.sample_rate), mode="constant")
 
             # Trim excess silences to compensate for gaps in spectrograms (issue #53)
             generated_wav = encoder.preprocess_wav(generated_wav)
-            
+
             # Play the audio (non-blocking)
             if not args.no_sound:
+                import sounddevice as sd
                 try:
                     sd.stop()
                     sd.play(generated_wav, synthesizer.sample_rate)
@@ -211,15 +195,15 @@
                     print("Continuing without audio playback. Suppress this message with the \"--no_sound\" flag.\n")
                 except:
                     raise
-                
+
             # Save it on the disk
             filename = "demo_output_%02d.wav" % num_generated
             print(generated_wav.dtype)
             sf.write(filename, generated_wav.astype(np.float32), synthesizer.sample_rate)
             num_generated += 1
             print("\nSaved output as %s\n\n" % filename)
-            
-            
+
+
         except Exception as e:
             print("Caught exception: %s" % repr(e))
             print("Restarting\n")

demo_toolbox.py

@@ -1,43 +1,37 @@
+import argparse
+import os
 from pathlib import Path
+
 from toolbox import Toolbox
 from utils.argutils import print_args
-from utils.modelutils import check_model_paths
-import argparse
-import os
+from utils.default_models import ensure_default_models
 
 
 if __name__ == '__main__':
     parser = argparse.ArgumentParser(
-        description="Runs the toolbox",
+        description="Runs the toolbox.",
         formatter_class=argparse.ArgumentDefaultsHelpFormatter
     )
-    
+
     parser.add_argument("-d", "--datasets_root", type=Path, help= \
         "Path to the directory containing your datasets. See toolbox/__init__.py for a list of "
         "supported datasets.", default=None)
-    parser.add_argument("-e", "--enc_models_dir", type=Path, default="encoder/saved_models", 
-                        help="Directory containing saved encoder models")
-    parser.add_argument("-s", "--syn_models_dir", type=Path, default="synthesizer/saved_models", 
-                        help="Directory containing saved synthesizer models")
-    parser.add_argument("-v", "--voc_models_dir", type=Path, default="vocoder/saved_models", 
-                        help="Directory containing saved vocoder models")
+    parser.add_argument("-m", "--models_dir", type=Path, default="saved_models",
+                        help="Directory containing all saved models")
     parser.add_argument("--cpu", action="store_true", help=\
-        "If True, processing is done on CPU, even when a GPU is available.")
+        "If True, all inference will be done on CPU")
     parser.add_argument("--seed", type=int, default=None, help=\
         "Optional random number seed value to make toolbox deterministic.")
-    parser.add_argument("--no_mp3_support", action="store_true", help=\
-        "If True, no mp3 files are allowed.")
     args = parser.parse_args()
+    arg_dict = vars(args)
     print_args(args, parser)
 
-    if args.cpu:
-        # Hide GPUs from Pytorch to force CPU processing
+    # Hide GPUs from Pytorch to force CPU processing
+    if arg_dict.pop("cpu"):
         os.environ["CUDA_VISIBLE_DEVICES"] = "-1"
-    del args.cpu
 
-    ## Remind the user to download pretrained models if needed
-    check_model_paths(encoder_path=args.enc_models_dir, synthesizer_path=args.syn_models_dir,
-                      vocoder_path=args.voc_models_dir)
+    # Remind the user to download pretrained models if needed
+    ensure_default_models(args.models_dir)
 
     # Launch the toolbox
-    Toolbox(**vars(args))    
+    Toolbox(**arg_dict)