Llama Tokenizer Unexpectedly Producing Unknown Token

[rehaanahmad2013]

System Info

• transformers version: 4.29.2
• Platform: Linux-5.19.0-1027-aws-x86_64-with-glibc2.31
• Python version: 3.10.8
• Huggingface_hub version: 0.14.1
• Safetensors version: 0.3.1
• PyTorch version (GPU?): 2.0.1+cu118 (True)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)

Who can help?

@ArthurZucker @younesbelkada I am trying to use special tokens with the LlamaTokenizer in Transformers 4.31.0 and with certain configurations of input, the tokenizer is returning a token id of 0 corresponding to the unknown token. For example, I have added the special token "<REPR_END>", and if I pass that through the tokenizer to get [1, 32003] which is good. Additionally if I pass the word "inform" through the tokenizer, I get [1, 1871], which is also good.

However, if I pass "<REPR_END>inform" through the tokenizer, I get [1, 32003, 0] which does not make sense. If I try this exact same input in Transformers 4.29.2, I get [1, 32003, 1871] which is correct.

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

from transformers.models.llama.tokenization_llama import LlamaTokenizer

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-13b-hf", use_auth_token=...)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.add_tokens(['<TARGET_BEGIN>', '<TARGET_END>', '<REPR_BEGIN>', '<REPR_END>'], special_tokens=True)

print(tokenizer("<REPR_END>inform")

Expected behavior

I should expect to get the output [1, 32003, 1871] but I do not. I instead get [1, 32003, 0]

[ArthurZucker]

Hey! 👋🏻 Thanks for providing a reproduction script. I suspect that you do not have tokenizers installed, since when I use use_fast = True (which is the default if you have tokenizers) the issue is not present.
Now, this behaviour is expected:

• Quick fix, use legacy=True when initialising the tokenizer: tokenizer = LlamaTokenizer.from_pretrained("meta-llama/Llama-2-13b-hf", legacy = True)
• Other quick fix, use the fast tokenizer ( pip install tokenizers)

This is a very nice catch otherwise! The issue is that [in, form] should be the tokenization of inform, but when we use the hack around sentencepiece, we actually just output [inform] which is not recognised as a token. Also if you pass "<REPR_END> inform" the extra space is automatically strip by default. This is also gonna be fixed.

cc @Narsil I think I'll implement handling the add_dummy_prefix = False parameter. As pointed out somewhere else, our decoding function is also broken for Llama (it add extra spaces).

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.