Unsupervised Word Segmentation for Neural Machine Translation and Text Generation

Byte Pair Encoding for Python!

nfelib - bindings Python para e ler e gerir XML de NF-e, NFS-e nacional, CT-e, MDF-e, BP-e

Fast bare-bones BPE for modern tokenizer training

Subword Encoding in Lattice LSTM for Chinese Word Segmentation

Simple-to-use scoring function for arbitrarily tokenized texts.

BBPE 底层实现

Learning BPE embeddings by first learning a segmentation model and then training word2vec

Byte-Pair Encoding (BPE) (subword-based tokenization) algorithm implementaions from scratch with python

Subword-augmented Embedding for Cloze Reading Comprehension (COLING 2018)

A ridiculously fast Python BPE (Byte Pair Encoder) implementation written in Rust

Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenization [arXiv 2025]

This repository provides a clear, educational implementation of Byte Pair Encoding (BPE) tokenization in plain Python. The focus is on algorithmic understanding, not raw performance.

Free to use tokenizers trained on the Darija language.

Byte-Pair Encoding tokenizer for training large language models on huge datasets

Docker container to expose the OpenAI tokenizer as a REST service

Byte Pair Encoding (BPE)

Natural Language EnCoder-Decoder: word, char, bpe etc

A lightweight, from-scratch implementation of Byte Pair Encoding (BPE) tokenization in Python.