”never_split“ not working on BertTokenizer

[lllyyyqqq]

System Info

transformers 4.28.1
python 3.8.13

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

• I load BertTokenizer using my own vocab.txt, and add '[outline]' into never_split, which is included in my vocab.txt. However, '[outline]' got splitted. Following is my code:

tokenizer = BertTokenizer.from_pretrained(pretrained_path,never_split=['[outline]']) input = "。[outline]" print(tokenizer.tokenize(input)) # ['。', '[', 'out', '##line', ']']

• I also do:
  print(tokenizer.basic_tokenizer.tokenize(input)) #['。', '[', 'outline', ']']

Expected behavior

When I do:
tokenizer.tokenize("。[outline]")
Get the result as ['。', '[outline]'], the tokens in never_split don't be splited.

[amyeroberts]

cc @ArthurZucker @younesbelkada

[zspo]

The '[' or ']' in BertTokenizer is punctuation, it will be split at first. And the outline or [outline] is not in vocab, its will be set UNK. It doesn't seem to make sense anymore.
Look the code: https://github.com/huggingface/transformers/blob/main/src/transformers/models/bert/tokenization_bert.py#L446

[lllyyyqqq]

The '[' or ']' in BertTokenizer is punctuation, it will be split at first. And the outline or [outline] is not in vocab, its will be set UNK. It doesn't seem to make sense anymore. Look the code: https://github.com/huggingface/transformers/blob/main/src/transformers/models/bert/tokenization_bert.py#L446

Thanks for replying. As stated before, I am using my own vocab, and ’[outline]‘ is in it,

tokenizer = BertTokenizer.from_pretrained(my_vocab_path, never_split='[outline]')
print(tokenizer.convert_tokens_to_ids('[outline]'))
print(tokenizer.convert_tokens_to_ids('。'))
print(tokenizer.tokenize('。[outline]'))

[ArthurZucker]

Hey, reading the doc for the BertTokenizer, you should be using the do_basic_tokenize=True argument, as mentioned here.

[lllyyyqqq]

Hey, reading the doc for the BertTokenizer, you should be using the do_basic_tokenize=True argument, as mentioned here.

Your link is broken, it says '404 - page not found'?
Plus, do_basic_tokenize=True is default setting. Even if I add it intentionally, the result stays the same.

tokenizer = BertTokenizer.from_pretrained(my_vocab_path, never_split=['[outline]'], do_basic_tokenize=True)
print(tokenizer.tokenize('。[outline]')) # ['。', '[', 'out', '##line', ']']

Correct me if I do anything wrong.

[ArthurZucker]

Sorry, anyway the argument was set to True by default so that's not the problem.
Let's me investigate, in the mean time doing tokenizer.add_token("[outline]", special_token = True)" should (I think) prevent it from being split

[ArthurZucker]

( the doc mentions :

 never_split (`List[str]`, *optional*)
                Kept for backward compatibility purposes. Now implemented directly at the base class level (see
                [`PreTrainedTokenizer.tokenize`]) List of token not to split.

[ArthurZucker]

The best solution is to add the token to the list of special tokens using the add_token method

[lllyyyqqq]

Yeah, add it as special_token does take care of the splitting problem. But in the latter process, I will decode with argument skip_special_tokens=True. Then the token will be skipped, while I don't want it be. For now, I add it to the special token list, but I still suggest fixing the never_split argument.

[ArthurZucker]

Then that means that the token that you want to add is not special. I think that if you add it without the special_token set to True it should not be spilt no?

[lllyyyqqq]

Without special_token set to True, it will be splitted.

[ArthurZucker]

No it won't :

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased", use_fast=False)
tokenizer.add_tokens("[outline]")
tokenizer.added_tokens_encoder
>>>  {'[outline]': 30522}

tokenizer.encode("[outline]")
>>> [101, 30522, 102]
tokenizer.decode(tokenizer.encode("[outline]"))
>>> '[CLS] [outline] [SEP]'
print(tokenizer.tokenize(". [outline]"))
>>> ['.', '[outline]']

tokenizer.decode(tokenizer.encode(". [outline]"), skip_special_tokens=True)
>>> '. [outline]'

[lllyyyqqq]

In your case, it won't. But I am using a different vocab.txt, it splits.

[lllyyyqqq]

Seems like '[outline]' will not be added anyway, since it's already in the vocab.

[ArthurZucker]

I don't understand. You have a very specific usage, where you don't want to split [outline] that is already in your vocab.
The basic tokenizer works as expected: tokenizer.basic_tokenizer.tokenize("[outline]") will not split it.
When you are calling tokenize on the BertTokenizerClass the _tokenize function is then called, which relies on the all_special_ids. That means that the token should be added to both lists.

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased", use_fast=False, never_split= ["outline"])
tokenizer.add_tokens("[outline]")

I am guessing that this should work