GitHub - mdabir1203/Rust_Tokenizer_BPE: Byte-Pair Algorithm implementation (Karpathy version of Rust) · GitHub

Name		Name	Last commit message	Last commit date
src		src
target		target
tests		tests
Cargo.lock		Cargo.lock
Cargo.toml		Cargo.toml
README.md		README.md

Repository files navigation

A Rust implementation of the Karpathy implemented BPE Tokenizer Algorithm [[ WIP ]]

About

Byte-Pair Algorithm implementation (Karpathy version of Rust)

bpe karpathy llm tokenizer-nlp

0 stars

1 watching

Report repository

Releases

Packages

Used by

Contributors

Languages