mT5 additional_special_tokens seems not work

[PiggyFan]

I want add some special tokens such as <CON_START> . But T5tokenizer/MT5tokenizer both can't tokenize correctly after using additional_special_tokens parameter. It still split these special tokens to subwords.

It works when using OpenAIGPTTokenizer additional_special_tokens parameter. It's clear that after declare additional_special_tokens parameter, OpenAIGPTTokenizer tokenize as one word rather split it.

The version of transformers is 4.2.2
And I'm not sure this problem is related with issue624 in T5 which talk about SentencePiece extra vocab.

Thank you for your feedback

[LysandreJik]

Hi! Could you either post a link to your notebook or your code as actual code? Images makes it impossible to copy/paste or for others to search with similar issues. Thanks.

[PiggyFan]

Hi! Could you either post a link to your notebook or your code as actual code? Images makes it impossible to copy/paste or for others to search with similar issues. Thanks.

Here is colab notebook link. Thanks for your reply.

https://colab.research.google.com/drive/1fbp7VvnUvbf5r8CSitOg2pDZyM47y9xj?usp=sharing

[LysandreJik]

Hi! Indeed, this is a bit misleading. Special tokens are considered as those that were in the pre-training, that is: unknown tokens, bos tokens, eos tokens, etc.

If you want to use special tokens that you use as special tokens, I would argue it is better to define them as simple tokens. Therefore doing the following:

>>> from transformers import MT5Tokenizer, MT5ForConditionalGeneration
... import torch
... special_tokens = ['<POS>', '<NEG>','<CON_START>','<START>','<END>'] # Set the special tokens
... mt5_add_tokenizer = MT5Tokenizer.from_pretrained("google/mt5-small")
... mt5_add_tokenizer.add_tokens(special_tokens)
... print(mt5_add_tokenizer.tokenize("<POS> <CON_START> the biscuits and gravy were . <START>"))

You'll get the following output:

['<POS>', '<CON_START>', '▁the', '▁b', 'iscuit', 's', '▁and', '▁grav', 'y', '▁were', '▁', '.', '<START>']

Let me know if that makes sense.

[PiggyFan]

Thank you very much, it works well.
I was confused by issue5940 which mentioned "special tokens are carefully handled by the tokenizer (they are never split)". If use add_token() method, the additional simple tokens may still split?
In addition, the code in Colab notebook as above link, OpenaiTokenizer should use add_tokens method rather than add_special_tokens (define them as a simple tokens) ?

[LysandreJik]

For tokens that cannot be identified as being either:

• eos_token
• bos_token
• cls_token
• unk_token
• pad_token
• sep_token
• mask_token

Then I would recommend using the add_token method. These tokens shouldn't split either:

>>> from transformers import BertTokenizer
>>> tokenizer = BertTokenizer.from_pretrained("bert-base-cased")
>>> tokenizer.add_tokens(["<CON_START>", "<CON_ST"])
2
>>> tokenizer.tokenize("<CON_START>")
['<CON_START>']
>>> tokenizer.tokenize("<CON_STAR>")
['<CON_ST', 'AR', '>']

[PiggyFan]

Thank for your help and recommend.

[Victordongy]

For tokens that cannot be identified as being either:

• eos_token
• bos_token
• cls_token
• unk_token
• pad_token
• sep_token
• mask_token

Then I would recommend using the add_token method. These tokens shouldn't split either:

>>> from transformers import BertTokenizer
>>> tokenizer = BertTokenizer.from_pretrained("bert-base-cased")
>>> tokenizer.add_tokens(["<CON_START>", "<CON_ST"])
2
>>> tokenizer.tokenize("<CON_START>")
['<CON_START>']
>>> tokenizer.tokenize("<CON_STAR>")
['<CON_ST', 'AR', '>']

Hi it seems add_tokens not working with AutoTokenizer , but work with specific-defined tokenizer like BertTokenizer,

special_tokens = ["-Title-"]
#tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')
tokenizer.add_tokens(special_tokens)
tokenizer.tokenize("-Title-")
# ['-', 'title', '-']

[ArthurZucker]

He! Thanks for reporting. This should be fixed by #23909