Open
Description
Try to look at rust gems BPE tokenizer performance and figure out if it is worth adapting it in the ML tokenizers library.
https://github.com/github/rust-gems/blob/main/crates/bpe/README.md
Try to look at rust gems BPE tokenizer performance and figure out if it is worth adapting it in the ML tokenizers library.
https://github.com/github/rust-gems/blob/main/crates/bpe/README.md