Can I train the Chinese model?

[Tsangchi-Lam]

I want to train the Chinese model. Do you support mixed input in Chinese and English?

[Kreevoz]

Look at issue #41 to check the current progress.

[yl4579]

You can, but with the current PL-BERT in English the quality won’t be as good it’s originally proposed to be. I’m working on multilingual PL-BERT now and it may take one or two months to finish.

[yl4579]

See yl4579/StyleTTS#10 for more details.

[hermanseu]

@yl4579 I trained styletts2 successfully using Chinese data, it sound very good. As wavlm-base-plus only supporting English, I used a Chinese Hubert model as SLM. When I want to train a model both for Chinese and English, I can not find a pre-trained model sopport Chinese and English at the same time. About SLM，Do you have any suggestions ?

[yl4579]

You can try whisper encoder that was trained with multiple languages. You can also try multilingual wav2vec2.0: https://huggingface.co/facebook/wav2vec2-large-xlsr-53

[zhouyong64]

@yl4579 I trained styletts2 successfully using Chinese data, it sound very good.

Did you use the English PL-BERT or did you train PL-BERT with Chinese data?

[hermanseu]

train PL-BERT with Chinese data

[Moonmore]

I trained styletts2 successfully using Chinese data, it sound very good. As wavlm-base-plus only supporting English, I used a Chinese Hubert model as SLM. When I want to train a model both for Chinese and English, I can not find a pre-trained model sopport Chinese and English at the same time. About SLM，Do you have any suggestions ?

What is your modeling unit? IPA or Pinyin?

[hermanseu]

@Moonmore The modeling unit is pinyin.

test.zip is a synth sample.

[zhouyong64]

@Moonmore The modeling unit is pinyin.

test.zip is a synth sample.

Do you use the tone of pinyin when training Chinese PL-BERT? I believe StyleTTS uses F0 for Chinese tones. Can this PL-BERT with tones work with StyleTTS?

[hermanseu]

I trained Chinese PL-BERT without pinyin tones. But maybe PL-BERT with tones will also work normally, so you can try.

[zhouyong64]

I trained Chinese PL-BERT without pinyin tones. But maybe PL-BERT with tones will also work normally, so you can try.

How many samples did you use to train Chinese PL-BERT?

[hermanseu]

@zhouyong64 I used about 84,000,000 text sentences to train the Chinese PL-BERT model.

[Moonmore]

@Moonmore The modeling unit is pinyin.

test.zip is a synth sample.

Sounds really good. I would like to ask if the pinyin unit you mentioned cannot be disassembled into phones? How to align plbert and text input?