Handle missing tokenizer_class

[baloodevil]

tokenizers.js expects tokenizer_class to exist in a tokenizer_config.json file according to this line. But a common model like bert-base-uncased does not have tokenizer_class. So the following code

import { AutoTokenizer } from "@xenova/transformers";
const tokenizer = await AutoTokenizer.from_pretrained("bert-base-uncased");
const { input_ids } = await tokenizer("I love transformers!");

throws error - TypeError: Cannot read properties of undefined (reading 'replace')

[xenova]

Hi there. Please use Xenova/bert-base-uncased as it provides an up-to-date tokenizer_config.json

[baloodevil]

That worked, thank you. The documentation page should be updated with that model name and add information on what models will work (if it doesn't exist elsewhere).

What is involved in getting a model to work as a tokenizer? Is it just converting to ONNX and making tokenizer.json and modifying tokenizer_config.json? I want to use this model, but it doesn't have tokenizer.json and tokenizer_config.json also doesn't have tokenizer_class.

[xenova]

The documentation page should be updated with that model name and add information on what models will work (if it doesn't exist elsewhere).

Oh yes, good spot! Feel free to open a PR if you'd like (or I can do it when I have some time). All that needs to be changed is this:
https://github.com/xenova/transformers.js/blob/46dd49064f681d017e5ff71a75c0b96e2d5686be/src/tokenizers.js#L10

Do get your model working, you can use our conversion tool: https://huggingface.co/docs/transformers.js/custom_usage#convert-your-models-to-onnx

1. Clone the repo
2. Install the requirements:

   pip install -r scripts/requirements.txt
   

3. Run the conversion command

   python -m scripts.convert --quantize --model_id medicalai/ClinicalBERT
   

[baloodevil]

Cloned and made doc changes, but forking your repo then pushing my changes gives error...

> git push -u origin main
To https://github.com/baloodevil/transformers.git
 ! [rejected]        main -> main (fetch first)
error: failed to push some refs to 'https://github.com/baloodevil/transformers.git'
hint: Updates were rejected because the remote contains work that you do
hint: not have locally. This is usually caused by another repository pushing
hint: to the same ref. You may want to first integrate the remote changes
hint: (e.g., 'git pull ...') before pushing again.
hint: See the 'Note about fast-forwards' in 'git push --help' for details.

I tried pulling first. If you know the solution to the git process, great, but if not, probably not worth your time to resolve just to get lines of docs updated.

Working on converting to ONNX now. It seems to be stopped at Ignore MatMul due to non constant B: /[/distilbert/transformer/layer.5/attention/MatMul_1]. I may need to update cuda from 11.6 or install TensorRT and cuBLAS if this is related.

Just saw your repo has Xenova/Bio_ClinicalBERT. I'll try those embeddings to see if they work for my use case.

[xenova]

Cloned and made doc changes, but forking your repo then pushing my changes gives error...

It looks like you're trying to push to my transformers fork (not transformers.js): https://github.com/baloodevil/transformers

Working on converting to ONNX now. It seems to be stopped at Ignore MatMul due to non constant B: /[/distilbert/transformer/layer.5/attention/MatMul_1].

The model isn't too large, so it shouldn't be a memory issue. If you want I can upload a converted checkpoint to the hub.

[xenova]

I've updated the docs with the correct tokenizer 👍

Working on converting to ONNX now. It seems to be stopped at Ignore MatMul due to non constant B: /[/distilbert/transformer/layer.5/attention/MatMul_1]

I converted it myself and it just takes a little long due to the large vocabulary size (>100k). On my computer it took around 15 minutes. For your convenience, I uploaded my version to the hub here, so feel free to use that instead of converting your own.