🚨🚨🚨 [SPM] Finish fix spm models 🚨🚨🚨

[ArthurZucker]

What does this PR do?

Modifies Llama and T5 other sentencepiece based tokenizer will follow.

Previous behaviour is always possible with tokenizer = AutoTokenizer.from_pretrained("huggyllama/llama-7b", legacy = True)

The goal of transformers's wrapping around sentencepiece

To clarify, we want to:

• be able to choose the behaviour of the special/added tokens. This means handling the stripping, encoding and decoding of such tokens
• allow users to easily add new tokens, with tokenenizer.add_tokens(...) instead of having to load the protobuf file, modify the vocab, save it and reload the sentencepiece processor.

The current and past problems with our wrappers

Let's use both T5 and Llama as reference models. Currently, we do not mimic the behaviour of adding words to the actual sentencepiece vocabulary. This is an issue for anyone expecting (and rightfully) that adding tokens does not modify the behaviour of the model.

Adding a word to sentencepiece's vocab

This can be done using: (source)

>>> # wget https://huggingface.co/huggyllama/llama-7b/resolve/main/tokenizer.model
>>> from sentencepiece import sentencepiece_model_pb2 as model
>>> import sentencepiece as spm
>>> sp_model = model.ModelProto()
>>> sp_model.ParseFromString(open('tokenizer.model', 'rb').read())
>>> token = "your_token"
>>> sp_model.pieces.add(piece=f"{token}",score=0.0,type=model.ModelProto.SentencePiece.USER_DEFINED,)
>>> with open('new.model', 'wb') as f:
...     f.write(sp_model.SerializeToString())

then load the sp_model:

>>> sp_model = spm.SentencePieceProcessor()
>>> sp_model.Load('new.model')

Then, try the following :

>>> sp_model.encode("your_tokenHello", out_type=str)
["_", "your_token", "Hello"]

Adding a word to a `PretrainedTokenizer

This can be done using tokenizer.add_tokens(["your_token"]). It is a lot simpler indeed.
But the output you will get is:

>>> from transformers import AutoTokenizer
>>> tokenizer = AutoTokenizer.from_pretrained("huggyllama/llama-7b", legacy = True, use_fast = False)
>>> tokenizer.add_tokens(["your_token"])
>>> tokenizer.tokenize("your_tokenHello")
["your_token", "_Hello"]

>>> tokenizer = AutoTokenizer.from_pretrained("huggyllama/llama-7b", legacy = False, use_fast = False)
>>> tokenizer.add_tokens(["your_token"])
>>> tokenizer.tokenize("your_tokenHello")
["your_token", "Hello"]

This is because we always split the text on the added tokens, and give the text on the left and right to the sentencepiece model. But, most sentencepiece models add a prefix space _ (the SPIECE_UNDERLINE character). Thus, when the transformers tokenizers splits "your_tokenHello", it encode your_token with the tokenizer.added_tokens_encoder and thus does not add a prefix space, and then encode Hello with the sentencepiece model, which adds a prefix space and thus outputs _Hello.

Other missmatches:

# t5-base tokenizer
>>> tokenizer.encode("<extra_id_0>. Hello", add_special_tokens = False)
[32099, 3, 5, 8774] # ['<extra_id_0>', ' ▁', '.', '▁Hello']
# seqio.SentencePieceVocabulary(vocab_path, extra_ids = 300)
>>> processor.encode("<extra_id_0>. Hello")
[32099, 5, 8774] # ['<extra_id_0>', '.', '▁Hello']

TLDR; this shows the only way we can actually and properly handle added tokens and sentencepiece. We have to disable automatic prefix addition, and always encode with a token that is part of the vocab at the beginning to properly encode the first token, whether it has a prefix space or not. Yes this is dirty and sad, but the previous fix was removing the extra space, which was cleaner but had a corner cases #25176.

The same issue happens with fast tokenizers:

>>> from transformers import AutoTokenizer
>>> tokenizer = AutoTokenizer.from_pretrained("huggyllama/llama-7b", use_fast = True)
>>> tokenizer.add_tokens(["your_token"])
>>> tokenizer.tokenize("your_tokenHello")
["_your_token", "Hello"]

>>> tokenizer.add_tokens(["your_token_special"], True)
>>> tokenizer.tokenize("your_token_specialHello")
['your_token_special', '▁Hello']

Another issue 😈

So, here, the issue is that before the special token, even if there is no rstrip or lstrip (both are set to False), we have very strange behaviours:

>>> tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf", use_fast = True)

>>> tokenizer.tokenize("<s>inform<s>")
# prefix space is eaten
['<s>', '▁inform', '<s>']

>>> tokenizer.tokenize("<s>inform <s>")
# prefix space is not eaten for the second <s>
['<s>', '▁inform', '▁', '<s>']

>>> tokenizer.tokenize(" <s>inform <s>")
# prefix space is not eaten for the second <s>
['▁▁', '<s>', '▁inform', '▁', '<s>']

>>> tokenizer.tokenize(" <s>inform <s> ")
# prefix space is not eaten for the first <s>, extra space added (known)
['▁▁', '<s>', '▁inform', '▁', '<s>', '▁▁']

>>> tokenizer.tokenize("inform <s> ")
# prefix space is added to inform
['▁inform', '▁', '<s>', '▁▁']

Note that tokenizer.convert_tokens_to_ids("▁▁") = 259 while tokenizer.convert_tokens_to_ids("▁") = 29871
Also if we add a prefix space to special tokens the beginning, we are probably gonna break a lot of things

[ArthurZucker]

Help for reviewers

[ArthurZucker]

The previous test values were not really good, with this update it makes more sense

[ArthurZucker]

Manually add the _ (spiece underline)

[ArthurZucker]

That's the Hack:

• all spm models have a tokenizer. Whether or not it is in the sentencepiece vocab does not matter.
• we need to do this because since add_dummy_prefix = False the sentencpiece model always ALWAYS strips any SPIECE_UNDERLINE. So sp_model.encode(SPIECE_UNDERLINE + "Hello" , out_type=str) will give [Hel,llo] instead of [_Hel, llo].
• previously, we removed added extra space. This is okay, but fails for words that should be split like inform. What happened before was that we would tokenize as _inform then remove _ and we have inform. But, the actual tokenization of inform is in,form and inform is not part of the vocab!

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[sgugger]

This makes sense to me. As usual with breaking changes, can you put three 🚨 in the title and show in the description how to enable the past behavior for users who want it?

[amyeroberts]

LGTM! Thanks for working on this v. tricky problem 🤗

[ArthurZucker]

Will fix the prefixing of special tokens!

[faaany]

@ArthurZucker any update to this PR?

[ArthurZucker]

Hey @faaany, I am updating it right now!

[ArthurZucker]

Reverted the changes as adding proper support for add_prefix_space is actully questionable. The usecase is already wrong as you should be reverse looking for ids not strings. See #24846 (adding prefix space was almost never done properly as the decoders were not updated as well)

[ArthurZucker]

pinging @sgugger for a final review !

[sgugger]

LGTM! Would be nice to make the two tests skipped smaller so that they pass.

[ArthurZucker]

Will do so in a follow up PR!

[zhacmsra]

I have added the legacy=True as "enc = AutoTokenizer.from_pretrained(model_path, legacy = True, use_fast=False)" but I have still gotten an error, which is a "Not a string" error. Anyone can give a hint what is going on here?

[ArthurZucker]

@zhacmsra the issue is in loading the vocabulary file, not 100% sure it's related to this. Can you open a new Issue with a reproducer please?

[zhacmsra]

Hi Arthur, you are correct. I figured out that it is not related to this PR. The networking problem broke the input model and result in error inputs. Sorry for this trouble. Thank you for the kind and timely response.