Unable to Load Custom GPT2 Tokenizer - " data did not match any variant of untagged enum ModelWrapper at line 1 column 3193814" Error

[maghwa]

I am encountering an issue when trying to load a custom merged GPT2 tokenizer using GPT2TokenizerFast. Despite ensuring that the tokenizer.json file is correctly formatted, I receive the following error:
data did not match any variant of untagged enum ModelWrapper at line 1 column 3193814

tokenizer-2.json

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[ArthurZucker]

Hey ! Having a look right now!

[ArthurZucker]

Ok, one thing is that your tokenizer seem to have 2 pre_tokenizers, not wrapped in a sequence that is super weird 😅 how did you merge?

[pietrolesci]

Hi @ArthurZucker,
I am encountering the same problem. I am training using the Rust library directly (since I need to log all possible merges and their counts, re #1523) and would like to load the tokenizer from transformers. Below a simple snippet that describes what I am doing. Am I missing something?

fn main() -> Result<()> {

    let mut trainer = BpeTrainerBuilder::new()
        .show_progress(true)
        .vocab_size(2000)
        .min_frequency(2)
        .special_tokens(vec![
            AddedToken::from(String::from("|endoftext|"), true),
        ])
        .build();

    let mut new_tok: TokenizerImpl<BPE, NormalizerWrapper, ByteLevel, ByteLevel, ByteLevel> = TokenizerBuilder::new()
        .with_model(BPE::default())
        .with_pre_tokenizer(Some(ByteLevel::new(false, true, true)))
        .with_post_processor(Some(ByteLevel::default()))
        .with_decoder(Some(ByteLevel::default()))
        .build()?;

    new_tok.train_from_files(&mut trainer, vec!["data/test_small.txt".to_string()])?;

    new_tok.save("tokenizer.json", true)?;

tok = PreTrainedTokenizerFast(tokenizer_file="tokenizer.json")
# >>> Exception: data did not match any variant of untagged enum ModelWrapper at line 1100 column 3

P.S. I pulled tokenizers and transformers both from main.

[ArthurZucker]

Mmm transformers does not really support tokenizers==0.20.0 yet, so that's one thing that is weird. Could you make sure that you are using the latest version of tokenizers ( the error should now be much more intuitive thanks to #1595 and the likes of it!

[pietrolesci]

HI @ArthurZucker,
Thanks a lot for the swift reply. I think the problem is compatibility between transformers and tokenizers: PreTrainedTokenizerFast does not know how to parse tokenizer.json. Importantly, it knows how to parse a tokenizer object! So to fix it by:

1. installing the python bindings from source which allows me to use the updated rust code in python (btw, the process to install from source was super easy, I cd'ed into bindings/python and simply pip install -e .; the updates I made to the rust backend are reflected in python -- fantastic 🚀 ).
2. Reading the tokenizer.json using the from tokenizers import Tokenizer
3. Passing the tokenizer object to PreTrainedTokenizerFast

That is,

from tokenizers import Tokenizer
from transformers import PreTrainedTokenizerFast

backend_tok = Tokenizer.from_file("modified_bpe/tokenizer.json")
tok = PreTrainedTokenizerFast(tokenizer_object=backend_tok)

[ArthurZucker]

I think using AutoTokenizer should be enough to load any fast tokenizers otherwise! 🤗

[Vectorrent]

I am currently running into the same problem with a custom tokenizer:

Error loading tokenizer: data did not match any variant of untagged enum ModelWrapper at line 40010 column 3

Oddly, this only happens in a Kaggle notebook, but not on my desktop. Both environments are using:

transformers==4.47.1 
tokenizers==0.21.0

Even more bizarre, is the fact that the tokenizer can be loaded and used when trained for 100k steps - but fails when trained for 10m steps!

I would greatly appreciate any advice you may have. My tokenizer can be re-produced with the following script:

import os
from itertools import islice
from typing import List, Union

from datasets import load_dataset
from tokenizers import (
    Tokenizer,
    decoders,
    models,
    normalizers,
    pre_tokenizers,
    processors,
    trainers,
)
from transformers import PreTrainedTokenizerFast

num_examples = 10_000_000

vocab_size = 8192
dropout = 0.1

save_path = "data/praxis"

pad_token = "<|pad|>"
bos_token = "<|bos|>"
eos_token = "<|eos|>"
start_token = "<|im_start|>"
end_token = "<|im_end|>"

dataset = load_dataset(
    "HuggingFaceFW/fineweb-edu",
    name="sample-100BT",
    split="train",
    streaming=True,
    trust_remote_code=True,
    cache_dir="data/datasets",
).shuffle(
    seed=59,
    buffer_size=10_000,
)

column = "text"
iterator = islice((item[column] for item in dataset), num_examples)

tokenizer = Tokenizer(
    models.BPE(dropout=dropout, cache_capacity=4096, byte_fallback=True)
)

tokenizer.add_special_tokens(
    [
        pad_token,
        bos_token,
        eos_token,
        start_token,
        end_token,
    ]
)

trainer = trainers.BpeTrainer(
    vocab_size=vocab_size,
    initial_alphabet=pre_tokenizers.ByteLevel.alphabet(),
    show_progress=True,
    special_tokens=[
        pad_token,
        bos_token,
        eos_token,
        start_token,
        end_token,
    ],
)

tokenizer.pre_tokenizer = pre_tokenizers.Sequence(
    [
        pre_tokenizers.WhitespaceSplit(),
        pre_tokenizers.ByteLevel(add_prefix_space=True, use_regex=True),
    ]
)

tokenizer.normalizer = normalizers.NFKC()

tokenizer.decoder = decoders.ByteLevel()
tokenizer.post_processor = processors.ByteLevel(trim_offsets=True)

tokenizer.train_from_iterator(iterator=iterator, trainer=trainer, length=num_examples)

trained_tokenizer = PreTrainedTokenizerFast(tokenizer_object=tokenizer)
trained_tokenizer.add_special_tokens(
    {
        "pad_token": pad_token,
        "bos_token": bos_token,
        "eos_token": eos_token,
        "additional_special_tokens": [start_token, end_token],
    }
)

os.makedirs(save_path, exist_ok=True)

trained_tokenizer.save_pretrained(save_path)

Update: some things I've tried:

• setting dropout to 0 does not fix the problem
• changing the seed does not work
• reducing 10m steps to 5m DOES fix the problem

[zengqingfu1442]

update transformers to the latest.

[ArthurZucker]

Or just update tokenizers==0.21.1. We made a breaking change which is not forward compatible, so if the tokenizers was saved with a more recent version of tokenizers, you won't be able to load it. This was done in 6a5fce9. Closing as it's expected! 🤗