Update added tokens

[ArthurZucker]

What does this PR do?

Fixes #1334 and refactors the AddedVocabulary. Previously the when a token was already part of the vocabuary it was not added to the added_tokens_map but still added to added_tokens_map_r which is not consistent. Now it is added in both even if it already existed.

Here is a small snippet of what is now possible in python

>>> from tokenizers import AddedToken, Tokenizer
>>> token = AddedToken("HEY")
>>> tokenizer = Tokenizer.from_pretrained("gpt2")
>>> tokenizer.get_added_tokens_decoder()

>>> tokenizer.add_tokens([token])
1
>>> token.special = False
>>> tokenizer.add_tokens([token])
1 
>>> tokenizer.get_vocab_size()
50258
>>> content = tokenizer.decode([4])
'%'
>>> tokenizer.add_tokens([AddedToken('%')])
1
>>> tokenizer.get_added_tokens_decoder()
{4: AddedToken("%", rstrip=False, lstrip=False, single_word=False, normalized=True, special=False),
 50256: AddedToken("<|endoftext|>", rstrip=False, lstrip=False, single_word=False, normalized=True, special=True),
 50257: AddedToken("HEY", rstrip=False, lstrip=False, single_word=False, normalized=True, special=False)}
>>> tokenizer.get_vocab_size()
50258

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[Narsil]

Why not use max(vocab_id) instead?

[ArthurZucker]

To account for potential holes 😢

[ArthurZucker]

And if we use max, we have to convert u32 to usize or return usize

[Narsil]

LGTM.

As said internally, I didn't have time to fully review, but most of my initial worries are addressed:

• is_special_token -> special is making things more uniform.
• The test breaking change is also fixing some other subtle bug within transformers when playing with special tokens (and we're releasing a breaking version anyway).

The rest looks OK.