add VITS model

[hollance]

What does this PR do?

Adds the VITS model for text-to-speech, in particular to support the MMS-TTS checkpoints (which use the same model architecture but a different tokenizer).

Fixes # (issue)

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[hollance]

Notes about the tokenizer:

1. This is not the VITS tokenizer but the one for MMS-TTS.
2. The vocab doesn't have padding (or unknown) tokens in it, but uses token_id 0 for this. That breaks on the HF tokenizers because it will split the input text on the padding token, so if I set pad_token_id = 0 then the letters that token_id 0 corresponds to will disappear from the text.
3. To fix this issue, I'm adding <pad> and <unk> to the vocab, but then in the model we set such token_ids to 0 before feeding the input into the first layer. It's a bit hacky. Ideas for a nicer solution are appreciated.
4. The tokenizer also inserts an additional token_id 0 in between every token. No idea why but that's how it works.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

[hollance]

This is ready for a first review yet.

Two checkpoints are currently available:

• https://huggingface.co/Matthijs/mms-tts-eng
• https://huggingface.co/Matthijs/mms-tts-nld

Small usage example:

from transformers import VitsMmsTokenizer, VitsModel
import torch

tokenizer = VitsMmsTokenizer.from_pretrained("Matthijs/mms-tts-eng")
model = VitsModel.from_pretrained("Matthijs/mms-tts-eng")
    
inputs = tokenizer(text="Hello, my dog is cute", return_tensors="pt")

outputs = model(inputs["input_ids"])
speech = outputs.audio

The current model is the MMS-TTS version, not the original VITS version. The conversion scripts can handle both, but for original VITS support the tokenizer is still missing.

Still needs to be done:

• tests
• tokenizer for actual VITS

@Vaibhavs10 For this review, in particular could you verify the names of the layers in the flow layers etc make sense? Thanks!

[Vaibhavs10]

Just reviewed the flow layer names for the coupling block and the duration predictor. Perfect!I Fantastic from a code PoV. Mega job @hollance 🤗

[sanchit-gandhi]

Looks really great already @hollance - nice work on simplifying such a new model for the lib. The high-level API works really nicely and the code by-and-large reads well, mostly just nits on my part

[sanchit-gandhi]

Hey @ArthurZucker - wonder if you know a more elegant workaround for this tokenization issue?

[ArthurZucker]

I think that the vocab should be updated in the convert_original_checkpoint, and just add the special tokens here.
I am not sure I understand the problem, they pad with 0 but 0 maps to "k". So they pad with "k"?

[hollance]

Yes, they pad with token_id 0. The problem is that the tokenizer splits the input on the padding token and tokenizes these subsequences separately, so anywhere there is a k in the text it will disappear.

For example, It stinks here gets tokenized as It stin and s here because k is not treated as a regular token but as padding.

[amyeroberts]

I'm not super familiar with the tokenization logic, so bear with me.

Am I right in understanding:

• The model is trained with 0 as a pad token
• The token_id 0 maps to "k" in the vocabulary
• The tokenizer first splits on the pad token, and then tokenizes
• Remapping the padding token to "<pad>" means we keep "k" in our input sequences
• We filter any token_ids corresponding to "<pad>" in our model? How is this done - setting them back to 0? With a mask?

Now, what happens if I have the input sentence "I like the letter k"?. When k is mapped to token_id 0, as the model was trained with this as a pad token, will it interpret k as padding?

[ArthurZucker]

I think @amyeroberts raised very valid questions.

• if they pad with k then k is the padding token non?
• if <pad> and k both point to 0, we don't really have a way in transformers to make sure this can be done. But I don't see why it should be done since they pad with k!
• if the problem is that our tokenize methods splits on the special tokens, including padding, but you don't want this to happen, then you can just overwrite tokenize. Given that this model does not seem to use bos nor eos you can probably ignore the lstrip and rstip logics, and just take what you have in _tokenize.

[sanchit-gandhi]

Now answering @ArthurZucker's points!

if they pad with k then k is the padding token non?

Explained above - they pad with the 0 token, which also corresponds to k in the vocab

if <pad> and k both point to 0, we don't really have a way in transformers to make sure this can be done

It is indeed the case that both <pad> and k point to 0 - hence the hackiness with the current tokenizer file to make this work, since there isn't a natural way of doing this in transformers

Probably then we have to keep a hacky workaround like this to make it work?

[sanchit-gandhi]

Wondering if you had any thoughts here @ArthurZucker @amyeroberts for the conflicting <pad>/k token both being mapped to 0? Explained a bit above why this works.

[ArthurZucker]

Can't we just use the padding token set to k?

[ArthurZucker]

Ah we can't otherwise k will be skipped when decoding without special and issue can arise. my bad I'll think about it, but a hack is most probably needed EDIT: actually when you decode, k will always be skipped if you set a padding token that maps to 0. Which is why I don't really think we should look into this too much. Either we keep a very simple hack, where in the vocab we make k and <pad> match to 0, and just set the padding token to <pad> ( this will work without further hack, and let's not check if it is or not in the vocab, because if somone want to change it by adding a new token it will not be in the vocab) or we just set the padding token to k.

[sanchit-gandhi]

Gone with this proposed 'hack' - we force the pad token at word boundaries, and then encode any k's to their appropriate id's. Note also that we are unlikely to ever need to decode input_ids with a TTS model - we put the input_ids in to the model, and get speech out.

[ArthurZucker]

Just a small review since @sanchit-gandhi left lots of comments. Feel free to ping me again!
(For next review let's get rid of single letter variables)

[ArthurZucker]

I think that the vocab should be updated in the convert_original_checkpoint, and just add the special tokens here.
I am not sure I understand the problem, they pad with 0 but 0 maps to "k". So they pad with "k"?

[ArthurZucker]

I think we can split this by defining intermediate variable like input_derivatives + input_derivatives_plus_one - 2 * input_delta which is re-used twice, same for (inputs - input_cumheights). Also let's remove single letter variables, make the math happening here more understandable! 😉

[hollance]

Some of the MMS-TTS checkpoints require the use of the tool uromanize from https://github.com/isi-nlp/uroman to convert the input script into the Latin alphabet. Since this is a separate Perl script, it is not included in Transformers and the user will have to run uromanize.pl themselves before using the tokenizer.

[sgugger]

I'm not too sure why I'm asked for a review here as all comments from @sanchit-gandhi are being ignored. The goal of integrating a model into Transformers is not to copy the original code without adapting it.

Please let's have all comments addressed before asking for the next round of reviews.

[hollance]

I'm not too sure why I'm asked for a review here as all comments from @sanchit-gandhi are being ignored.

No they aren't?! I've integrated most of his suggestions and replied with counterarguments otherwise.

[sanchit-gandhi]

Just a few more variable names to go - commented regarding the design decisions from the last review directly inline on the comments above

[sanchit-gandhi]

LGTM @hollance - last question from my side is whether we've pinpointed the changes between the original tokenizer and MMS tokenizer? If they're subtle enough that we can use one tokenizer for both models, we can re-name the tokenizer from VitsMmsTokenizer to VitsTokenizer (as suggested by @sgugger) and include a flag to toggle between the two padding behaviours?

[hollance]

Tokenizer can now handle both the original VITS models (which require phonemization) and the MMS-TTS models.

[sanchit-gandhi]

All points addressed so ready for a final review @amyeroberts. Thanks for your in-depth reviews here - the PR looks in pretty good shape!

[amyeroberts]

Amazing work - thanks for iterating on this and adding this model!

Only thing that needs to be addressed before merging are the slow tests to reflect tokenizer changes. Once that's done we're good to merge! 🥳

[sanchit-gandhi]

Hey @amyeroberts - to have full compatibility with the text-to-audio pipeline class, we need to indicate the sampling_rate of the predicted audio waveforms in the model config:

transformers/src/transformers/pipelines/text_to_audio.py

Line 82 in 2be8a90

sampling_rate = config.get(sampling_rate_name, None)

The sampling_rate corresponds to the sampling rate of the target audio that the model was trained on. It is not possible to determine in any way other than from the value in the original config of the model. MMS TTS models use a sampling rate of 16kHz, VITS TTS models use a sampling rate of 22kHz, but otherwise their configs are the same. The user needs to have an idea of the sampling rate that the model generates in order to know what rate to playback the audio, otherwise this leaves them prone to silent errors. IMO adding it as an attribute of the main model class should suffice here:

transformers/src/transformers/models/vits/modeling_vits.py

Line 1374 in ff3b08c

self.sampling_rate = config.sampling_rate

Note that we cannot just add the sampling_rate to the config and not the modelling file, this is not allowed by the CI:
https://app.circleci.com/pipelines/github/huggingface/transformers/71815/workflows/c0[…]cb-a064-7c1019e03630/jobs/904373/parallel-runs/0/steps/0-116

cc @ylacombe

[sanchit-gandhi]

As discussed offline with @amyeroberts, we'll add it as an allowed attribute in the config checker: 8b01633

[ydshieh]

Hi @hollance Thank you for adding this model into transformers 🤗 .

There is a test failing

python3 -m pytest -v tests/models/vits/test_modeling_vits.py::VitsModelTest::test_initialization

I skip it on the main branch. Would you like to help us investigate this test if you have some bandwidth? Otherwise we can take this on our side too.

If you decide to take a look, you have to remove the following line

transformers/tests/models/vits/test_modeling_vits.py

Line 172 in ab8cba8

@unittest.skip("currently failing")

so the test will be collected and run by pytest.

Let me know :-) Thank you!