Tacotron2 + WaveRNN experiments

[erogol]

Tacotron2: https://arxiv.org/pdf/1712.05884.pdf
WaveRNN: https://github.com/erogol/WaveRNN forked from https://github.com/fatchord/WaveRNN

The idea is to add Tacotron2 as another alternative if it is really useful then the current model.

• Code boilerplate tracotron2 architecture.
• Train Tacotron2 and compare results (Baseline)
• Train TTS current model in a comparable size with T2. (Current TTS model has 7M and Tacotron2 has 28M parameters)
• Add TTS specific architectural changes to T2 and compare with the baseline.
• Train WaveRNN a vocoder on generated spectrograms
• Train a better stopnet. Stopnet sometimes misses the prediction that leads to unstable predictions. Maybe it is better to use a RNN as previous TTS version.
• Release LJspeech Tacotron 2 model. (soon)
• Release LJSpeech WaveRNN model. (https://github.com/erogol/WaveRNN)

Best result so far: https://soundcloud.com/user-565970875/ljspeech-logistic-wavernn

Some findings:

• Adding an entropy loss for the attention seems to improve the cases hard to learn the alignment. It forces network to learn more sparse and noise free alignment weights.

entropy = torch.distributions.Categorical(probs=alignments).entropy()
entropy_loss = (entropy / np.log(alignments.shape[1])).mean()
loss += 1e-4 * entropy_loss

Here is the alignment with entropy loss. However, if you keep the loss weight high, then it degrades the model's generalization for new words.

• Replacing Prenet with a BatchNorm version ehnace the performance quite a lot.
• A network with BN Prenet is harder to learn the attention. It looks like the network needs a level of noise onto autoregressive connection to relate encoder output to network output. Otwerwise, in teacher forcing mode, network does not need encoder output since it finds previous prediction frame enough to generate the next frame.
• Forward attention seems more robust to longer sequences and faster to align. (https://arxiv.org/abs/1807.06736)

[erogol]

Tacotron2 baseline implementation aligns after 35K iterations.

Also it gives better melspectrograms L1 loss compared to Tacotron1. Tacotron2 trained 150K iters and Tacotron trained 320K,

Tacotron1 loss: 0.34
Decoder loss: 0.3115
Postnet loss: 0.2985

[erogol]

Tacotron2 + TTS updates aligns much faster after 14K. It also reaches the same loss values of the baseline implementation which it gets after 100K

Here is the attention alignment after 14K

[erogol]

Here is the first WaveRNN based result.

https://soundcloud.com/user-565970875/wavernn-taco2

[erogol]

Here is a pocket article read by the mode WaveRNN + Tacotron2
Tacotron2 - 90k iterations
WaveRNN - 550K iterations
https://soundcloud.com/user-565970875/pocket-article-wavernn-and-tacotron2

[ZohaibAhmed]

@erogol Awesome results. Did you train tacotron2 with batch size of 32 and r=1? Was it done on multi-gpu or just one?

[G-Wang]

@erogol Fatcord's wavernn works very well with 10 bit audio as well, which helps to eliminate quite a bit of the background static.