Synpaflex french dataset tacotron2 and MB-MelGAN support

[samuel-lunii]

Note that the files of the SynPaFlex dataset have to be reorganized in an LJSpeech manner. I made a script to do this automatically, which is not present here. I can also share it if you want :)

[dathudeptrai]

@samuel-lunii Is this pull request complete?

[ZDisket]

@samuel-lunii Is there a trained model?

[samuel-lunii]

@dathudeptrai almost complete :)

• I forgot to include Synpaflex in tensorflow_tts/configs/tacotron2.py
• Still need to update README.md and to create a notebook similar to this one to prepare synpaflex data before preprocessing.

@ZDisket I have trained models for tacotron2 (150k) and MB-MelGAN (780k), but I am not sure where to include them ?

[samuel-lunii]

@dathudeptrai it is complete now

[dathudeptrai]

@samuel-lunii

I have trained models for tacotron2 (150k) and MB-MelGAN (780k), but I am not sure where to include them ?

You can make a google colab for the inference. The model should be downloaded from google drive then i will fork and copy your model to upload into Huggingface Hub :D.

[samuel-lunii]

@dathudeptrai
Here is a link to the google colab for inference, everything is there :)

[dathudeptrai]

@samuel-lunii Many thanks :D, i will review and merge this weekend :D

[dathudeptrai]

@samuel-lunii can you fix a failing check :D

[samuel-lunii]

@dathudeptrai here is a colab for inference with FS2 trained at 200k steps with SynPaFlex dataset. Durations have been exported with tacotron2 trained at 150k steps.

[samuel-lunii]

@dathudeptrai
I realized that you set the sampling rate to 24000 Hz instead of 22050 Hz as I used for preprocessing, in this colab cell.

Also, adding the piece of code below in this cell, before # vocoder part, allows for finding the end of the synthesized utterance from alignment data, even if the synthesized mel spectrogram is much longer than it should be :

  # find the end of the sentence according to alignment data
  final_text_index = alignment_history[0].shape[0]
  final_frame_index = 0
  for frame in np.swapaxes(alignment_history[0],0,1):
    max_index = np.where(frame == np.amax(frame))[0][0] 
    final_frame_index += 1
    if max_index == final_text_index - 1:
      break

and then replace
audio = vocoder_model.inference(mel_outputs)[0, :-remove_end, 0]
by :
audio = vocoder_model.inference(mel_outputs[:,:final_frame_index,:])[0, :-1, 0]

So Vocoder inference is only performed on actual speech :)

[dathudeptrai]

@samuel-lunii Many thanks, i will upload your model to huggingface hub soon :D