Skip to content

Consider Rust BPE tokenizer implementation #7336

Open
@tarekgh

Description

@tarekgh

Try to look at rust gems BPE tokenizer performance and figure out if it is worth adapting it in the ML tokenizers library.

https://github.com/github/rust-gems/blob/main/crates/bpe/README.md

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions