VitsTokenizer decode without special tokens produces odd results

[xenova]

System Info

• transformers version: 4.35.2
• Platform: Linux-6.1.58+-x86_64-with-glibc2.35
• Python version: 3.10.12
• Huggingface_hub version: 0.19.4
• Safetensors version: 0.4.1
• Accelerate version: not installed
• Accelerate config: not found
• PyTorch version (GPU?): 2.1.0+cu121 (False)
• Tensorflow version (GPU?): 2.15.0 (False)
• Flax version (CPU?/GPU?/TPU?): 0.7.5 (cpu)
• Jax version: 0.4.20
• JaxLib version: 0.4.20
• Using GPU in script?: no
• Using distributed or parallel set-up in script?: no

Who can help?

@ArthurZucker (tokenizers) @Vaibhavs10 @sanchit-gandhi (audio team)

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

>>> from transformers import AutoTokenizer
>>> tokenizer=AutoTokenizer.from_pretrained('facebook/mms-tts-eng')
>>> tokenizer.encode('hello world')
[0, 6, 0, 7, 0, 21, 0, 21, 0, 22, 0, 19, 0, 9, 0, 22, 0, 25, 0, 21, 0, 5, 0]
>>> tokenizer.decode(tokenizer.encode('hello world'), skip_special_tokens=False)
'hello world'
>>> tokenizer.decode(tokenizer.encode('hello world'), skip_special_tokens=True)
'el ol'
>>> tokenizer.decode(tokenizer.encode('abcdefghijklmnopqrstuvwxyz'), skip_special_tokens=True)
'bdfhjmoqsuwy'

From the last example, it looks like it's taking the even-positioned elements.

Expected behavior

[0, 6, 0, 7, 0, 21, 0, 21, 0, 22, 0, 19, 0, 9, 0, 22, 0, 25, 0, 21, 0, 5, 0], for which the tokenized version is:

['k', 'h', 'k', 'e', 'k', 'l', 'k', 'l', 'k', 'o', 'k', ' ', 'k', 'w', 'k', 'o', 'k', 'r', 'k', 'l', 'k', 'd', 'k']

should be decoded as 'hello world', or something more informative than 'el ol'.

[xenova]

I understand users won't typically use the decode method, but it's still quite odd behaviour 😅

[ArthurZucker]

👿 I'm using my joker for this on, it's on the audio team 🤭

[xenova]

Here's the culprit:

transformers/src/transformers/models/vits/tokenization_vits.py

Lines 225 to 228 in 1d77735

	def convert_tokens_to_string(self, tokens: List[str]) -> str:
	if self.add_blank and len(tokens) > 1:
	tokens = tokens[1::2]
	return "".join(tokens)

[sanchit-gandhi]

Thanks for reporting @xenova! The VITS MMS tokenizer is very much non-standard in the sense that it uses the same token id as the special pad token as well as a standard vocab token => this is something we probably should have changed when we integrated the model! Is there a case where you'd need to encode and subsequently decode the input text? Happy to take a look at a fix if so! Otherwise, we can add a disclaimer to the docs saying that tokenizer.decode(tokenizer.encode(input_text)) != input_text due to this behaviour

[sanchit-gandhi]

Copying a related discussion from Slack:

VB
23 days ago
Yes, it is the pad token: https://huggingface.co/facebook/mms-tts-eng/blob/main/tokenizer_config.json#L8
AFAIK, k represents the character boundary (to seperate one character from the other). Since the text is split and converted to phonemic representation later, this helps segregate it. each character has a seperate phonemic representation.
each phonemic representation creates it's respective mel-spec based on the duration prediction.
k is later on ignored in during attention as the mask is set to 1 i.e. do not attend to it for it.
https://huggingface.co/facebook/mms-tts-eng/blob/c71de0fe7204c83f1c10820a7d696d0b450048ba/vocab.json#L23
I think the reason for choosing k in case of MMS was to make it be able to generalise across the multitudes of language they trained on.

Joshua Lochner
23 days ago
Oh wow that’s so interesting 😂 Thanks!

Joshua Lochner
23 days ago
I also checked some of the other languages, and indeed they also set a pad token to be one of those characters the vocabulary.

Sanchit
23 days ago
Yes it’s a bit messy, but they essentially just picked one of the existing characters in the MMS tokenizer to act as the padding token (in this case k), and then mask it during the forward pass with the attention mask
In the MMS integration to transformers I used the same behaviour they did in the original repo (use k as the padding token). Probably what would be more elegant is:

1. Adding a new padding token to the MMS vocab (e.g. )
2. Re-sizing the embedding layer to account for this new padding token with randomly initialised weights
3. Mask out the embeddings that come from the padding token in the forward pass (as is done already)
   This would break the existing MMS weights though, so unfortunately can’t be done now, but would have been a better option in retrospect (edited)

Sanchit
23 days ago
(The solution I explained above is how they do it in VITS with a dedicated token)

Amy
23 days ago
There’s a related discussion here: #24085 (comment)

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.