Problem with VarianceAdaptor implementation

[rishikksh20]

Here :

FastSpeech2/modules.py

Line 48 in bd4c341

x = x + pitch_embedding

and then

FastSpeech2/modules.py

Line 50 in bd4c341

energy_prediction = self.energy_predictor(x)

But as per paper detail the input of Energy predictor should be output of length regulator not the output of pitch predictor.
See the fastspeech 2 diagram clearly input of Energy predictor is x output of length regulator without pitch component.
Actual code should be like:

 def forward(self, x, duration_target=None, pitch_target=None, energy_target=None, max_length=None):

        duration_prediction = self.duration_predictor(x)
        if duration_target is not None:
            x, mel_pos = self.length_regulator(x, duration_target, max_length)
        else:
            duration_rounded = torch.round(duration_prediction)
            x, mel_pos = self.length_regulator(x, duration_rounded)
        
        pitch_prediction = self.pitch_predictor(x)
        if pitch_target is not None:
            pitch_embedding = self.pitch_embedding(torch.bucketize(pitch_target, self.pitch_bins))
        else:
            pitch_embedding = self.pitch_embedding(torch.bucketize(pitch_prediction, self.pitch_bins))
  
        
        energy_prediction = self.energy_predictor(x)
        if energy_target is not None:
            energy_embedding = self.energy_embedding(torch.bucketize(energy_target, self.energy_bins))
        else:
            energy_embedding = self.energy_embedding(torch.bucketize(energy_prediction, self.energy_bins))

        x = x + pitch_embedding
        x = x + energy_embedding
        
        return x, duration_prediction, pitch_prediction, energy_prediction, mel_pos

[rishikksh20]

One more thing is here:

FastSpeech2/data/ljspeech.py

Line 98 in bd4c341

energy = np.linalg.norm(mel_spectrogram, axis=0)

you calculate energy as L2 norm of Mel Spectrogram frame but auther mentioned compute L2-norm of the amplitude of each STFT frame as the energy for STFT frame.
Means getting calculate L2 norm of magnitude output here:

FastSpeech2/audio/stft.py

Line 79 in bd4c341

return magnitude, phase

[rishikksh20]

@ming024 also could you upload training and validation error curve plots (tensorboard curves), I have also implemented FastSpeech 2 and just want to match my error plots with yours.

[ming024]

@rishikksh20 Thanks for your idea. I will correct the code. I am also trying to use different normalization approach to make to implementation more intuitive. Update soon.

[rishikksh20]

@ming024 Total number of parameters?

[ming024]

With the Postnet removed:

DataParallel(
  (module): FastSpeech2(
    (encoder): Encoder(
      (src_word_emb): Embedding(149, 256, padding_idx=0)
      (position_enc): Embedding(1001, 256)
      (layer_stack): ModuleList(
        (0): FFTBlock(
          (slf_attn): MultiHeadAttention(
            (w_qs): Linear(in_features=256, out_features=256, bias=True)
            (w_ks): Linear(in_features=256, out_features=256, bias=True)
            (w_vs): Linear(in_features=256, out_features=256, bias=True)
            (attention): ScaledDotProductAttention(
              (softmax): Softmax(dim=2)
            )
            (layer_norm): LayerNorm((256,), eps=1e-05, elementwise_affine=True)
            (fc): Linear(in_features=256, out_features=256, bias=True)
            (dropout): Dropout(p=0.2, inplace=False)
          )
          (pos_ffn): PositionwiseFeedForward(
            (w_1): Conv1d(256, 1024, kernel_size=(9,), stride=(1,), padding=(4,))
            (w_2): Conv1d(1024, 256, kernel_size=(1,), stride=(1,))
            (layer_norm): LayerNorm((256,), eps=1e-05, elementwise_affine=True)
            (dropout): Dropout(p=0.2, inplace=False)
          )
        )
        (1): FFTBlock(
          (slf_attn): MultiHeadAttention(
            (w_qs): Linear(in_features=256, out_features=256, bias=True)
            (w_ks): Linear(in_features=256, out_features=256, bias=True)
            (w_vs): Linear(in_features=256, out_features=256, bias=True)
            (attention): ScaledDotProductAttention(
              (softmax): Softmax(dim=2)
            )
            (layer_norm): LayerNorm((256,), eps=1e-05, elementwise_affine=True)
            (fc): Linear(in_features=256, out_features=256, bias=True)
            (dropout): Dropout(p=0.2, inplace=False)
          )
          (pos_ffn): PositionwiseFeedForward(
            (w_1): Conv1d(256, 1024, kernel_size=(9,), stride=(1,), padding=(4,))
            (w_2): Conv1d(1024, 256, kernel_size=(1,), stride=(1,))
            (layer_norm): LayerNorm((256,), eps=1e-05, elementwise_affine=True)
            (dropout): Dropout(p=0.2, inplace=False)
          )
        )
        (2): FFTBlock(
          (slf_attn): MultiHeadAttention(
            (w_qs): Linear(in_features=256, out_features=256, bias=True)
            (w_ks): Linear(in_features=256, out_features=256, bias=True)
            (w_vs): Linear(in_features=256, out_features=256, bias=True)
            (attention): ScaledDotProductAttention(
              (softmax): Softmax(dim=2)
            )
            (layer_norm): LayerNorm((256,), eps=1e-05, elementwise_affine=True)
            (fc): Linear(in_features=256, out_features=256, bias=True)
            (dropout): Dropout(p=0.2, inplace=False)
          )
          (pos_ffn): PositionwiseFeedForward(
            (w_1): Conv1d(256, 1024, kernel_size=(9,), stride=(1,), padding=(4,))
            (w_2): Conv1d(1024, 256, kernel_size=(1,), stride=(1,))
            (layer_norm): LayerNorm((256,), eps=1e-05, elementwise_affine=True)
            (dropout): Dropout(p=0.2, inplace=False)
          )
        )
        (3): FFTBlock(
          (slf_attn): MultiHeadAttention(
            (w_qs): Linear(in_features=256, out_features=256, bias=True)
            (w_ks): Linear(in_features=256, out_features=256, bias=True)
            (w_vs): Linear(in_features=256, out_features=256, bias=True)
            (attention): ScaledDotProductAttention(
              (softmax): Softmax(dim=2)
            )
            (layer_norm): LayerNorm((256,), eps=1e-05, elementwise_affine=True)
            (fc): Linear(in_features=256, out_features=256, bias=True)
            (dropout): Dropout(p=0.2, inplace=False)
          )
          (pos_ffn): PositionwiseFeedForward(
            (w_1): Conv1d(256, 1024, kernel_size=(9,), stride=(1,), padding=(4,))
            (w_2): Conv1d(1024, 256, kernel_size=(1,), stride=(1,))
            (layer_norm): LayerNorm((256,), eps=1e-05, elementwise_affine=True)
            (dropout): Dropout(p=0.2, inplace=False)
          )
        )
      )
    )
    (variance_adaptor): VarianceAdaptor(
      (duration_predictor): VariancePredictor(
        (conv_layer): Sequential(
          (conv1d_1): Conv(
            (conv): Conv1d(256, 256, kernel_size=(3,), stride=(1,), padding=(1,))
          )
          (relu_1): ReLU()
          (layer_norm_1): LayerNorm((256,), eps=1e-05, elementwise_affine=True)
          (dropout_1): Dropout(p=0.5, inplace=False)
          (conv1d_2): Conv(
            (conv): Conv1d(256, 256, kernel_size=(3,), stride=(1,), padding=(1,))
          )
          (relu_2): ReLU()
          (layer_norm_2): LayerNorm((256,), eps=1e-05, elementwise_affine=True)
          (dropout_2): Dropout(p=0.5, inplace=False)
        )
        (linear_layer): Linear(
          (linear_layer): Linear(in_features=256, out_features=1, bias=True)
        )
      )
      (length_regulator): LengthRegulator()
      (pitch_predictor): VariancePredictor(
        (conv_layer): Sequential(
          (conv1d_1): Conv(
            (conv): Conv1d(256, 256, kernel_size=(3,), stride=(1,), padding=(1,))
          )
          (relu_1): ReLU()
          (layer_norm_1): LayerNorm((256,), eps=1e-05, elementwise_affine=True)
          (dropout_1): Dropout(p=0.5, inplace=False)
          (conv1d_2): Conv(
            (conv): Conv1d(256, 256, kernel_size=(3,), stride=(1,), padding=(1,))
          )
          (relu_2): ReLU()
          (layer_norm_2): LayerNorm((256,), eps=1e-05, elementwise_affine=True)
          (dropout_2): Dropout(p=0.5, inplace=False)
        )
        (linear_layer): Linear(
          (linear_layer): Linear(in_features=256, out_features=1, bias=True)
        )
      )
      (energy_predictor): VariancePredictor(
        (conv_layer): Sequential(
          (conv1d_1): Conv(
            (conv): Conv1d(256, 256, kernel_size=(3,), stride=(1,), padding=(1,))
          )
          (relu_1): ReLU()
          (layer_norm_1): LayerNorm((256,), eps=1e-05, elementwise_affine=True)
          (dropout_1): Dropout(p=0.5, inplace=False)
          (conv1d_2): Conv(
            (conv): Conv1d(256, 256, kernel_size=(3,), stride=(1,), padding=(1,))
          )
          (relu_2): ReLU()
          (layer_norm_2): LayerNorm((256,), eps=1e-05, elementwise_affine=True)
          (dropout_2): Dropout(p=0.5, inplace=False)
        )
        (linear_layer): Linear(
          (linear_layer): Linear(in_features=256, out_features=1, bias=True)
        )
      )
      (pitch_embedding): Embedding(256, 256)
      (energy_embedding): Embedding(256, 256)
    )
    (decoder): Decoder(
      (position_enc): Embedding(1001, 256)
      (layer_stack): ModuleList(
        (0): FFTBlock(
          (slf_attn): MultiHeadAttention(
            (w_qs): Linear(in_features=256, out_features=256, bias=True)
            (w_ks): Linear(in_features=256, out_features=256, bias=True)
            (w_vs): Linear(in_features=256, out_features=256, bias=True)
            (attention): ScaledDotProductAttention(
              (softmax): Softmax(dim=2)
            )
            (layer_norm): LayerNorm((256,), eps=1e-05, elementwise_affine=True)
            (fc): Linear(in_features=256, out_features=256, bias=True)
            (dropout): Dropout(p=0.2, inplace=False)
          )
          (pos_ffn): PositionwiseFeedForward(
            (w_1): Conv1d(256, 1024, kernel_size=(9,), stride=(1,), padding=(4,))
            (w_2): Conv1d(1024, 256, kernel_size=(1,), stride=(1,))
            (layer_norm): LayerNorm((256,), eps=1e-05, elementwise_affine=True)
            (dropout): Dropout(p=0.2, inplace=False)
          )
        )
        (1): FFTBlock(
          (slf_attn): MultiHeadAttention(
            (w_qs): Linear(in_features=256, out_features=256, bias=True)
            (w_ks): Linear(in_features=256, out_features=256, bias=True)
            (w_vs): Linear(in_features=256, out_features=256, bias=True)
            (attention): ScaledDotProductAttention(
              (softmax): Softmax(dim=2)
            )
            (layer_norm): LayerNorm((256,), eps=1e-05, elementwise_affine=True)
            (fc): Linear(in_features=256, out_features=256, bias=True)
            (dropout): Dropout(p=0.2, inplace=False)
          )
          (pos_ffn): PositionwiseFeedForward(
            (w_1): Conv1d(256, 1024, kernel_size=(9,), stride=(1,), padding=(4,))
            (w_2): Conv1d(1024, 256, kernel_size=(1,), stride=(1,))
            (layer_norm): LayerNorm((256,), eps=1e-05, elementwise_affine=True)
            (dropout): Dropout(p=0.2, inplace=False)
          )
        )
        (2): FFTBlock(
          (slf_attn): MultiHeadAttention(
            (w_qs): Linear(in_features=256, out_features=256, bias=True)
            (w_ks): Linear(in_features=256, out_features=256, bias=True)
            (w_vs): Linear(in_features=256, out_features=256, bias=True)
            (attention): ScaledDotProductAttention(
              (softmax): Softmax(dim=2)
            )
            (layer_norm): LayerNorm((256,), eps=1e-05, elementwise_affine=True)
            (fc): Linear(in_features=256, out_features=256, bias=True)
            (dropout): Dropout(p=0.2, inplace=False)
          )
          (pos_ffn): PositionwiseFeedForward(
            (w_1): Conv1d(256, 1024, kernel_size=(9,), stride=(1,), padding=(4,))
            (w_2): Conv1d(1024, 256, kernel_size=(1,), stride=(1,))
            (layer_norm): LayerNorm((256,), eps=1e-05, elementwise_affine=True)
            (dropout): Dropout(p=0.2, inplace=False)
          )
        )
        (3): FFTBlock(
          (slf_attn): MultiHeadAttention(
            (w_qs): Linear(in_features=256, out_features=256, bias=True)
            (w_ks): Linear(in_features=256, out_features=256, bias=True)
            (w_vs): Linear(in_features=256, out_features=256, bias=True)
            (attention): ScaledDotProductAttention(
              (softmax): Softmax(dim=2)
            )
            (layer_norm): LayerNorm((256,), eps=1e-05, elementwise_affine=True)
            (fc): Linear(in_features=256, out_features=256, bias=True)
            (dropout): Dropout(p=0.2, inplace=False)
          )
          (pos_ffn): PositionwiseFeedForward(
            (w_1): Conv1d(256, 1024, kernel_size=(9,), stride=(1,), padding=(4,))
            (w_2): Conv1d(1024, 256, kernel_size=(1,), stride=(1,))
            (layer_norm): LayerNorm((256,), eps=1e-05, elementwise_affine=True)
            (dropout): Dropout(p=0.2, inplace=False)
          )
        )
      )
    )
    (mel_linear): Linear(
      (linear_layer): Linear(in_features=256, out_features=80, bias=True)
    )
  )
)

Number of FastSpeech2 Parameters: 24983121

While the number reported by the paper is 24M.

[dathudeptrai]

@ming024 do you have any predicted plot of mels/f0/energy ?

[ming024]

@dathudeptrai I just do some correction on my code according to @rishikksh20 's advice so I need a few days to rerun training. I will update once the training finished.

[ming024]

@dathudeptrai Here is my spectrogram with predicted F0/energy

Training set:

Validation set:

[dathudeptrai]

@ming024 can you share tensorboard ? i don't know if ur f0 is overfit or not :D base on the above image -_-