etok — Entropy Tokenizer

Single-file BPE tokenizer in C. Zero dependencies. Trains faster than SentencePiece. Supports plain text, CSV, code, and DNA/FASTA sequences natively.

gcc -O3 -march=native -ffast-math -fopenmp -o etok etok.c -lm

---

Quick start

# Train on text
./etok train --data corpus.txt --out vocab.json --vocab_size 4096 -v

# Train on DNA/FASTA (auto-detected, no flags needed)
./etok train --data genome.fasta --out dna_vocab.json --vocab_size 512 -v

# Encode
./etok encode --vocab vocab.json --text "the cat sat on the mat"

# Decode
./etok decode --vocab vocab.json --ids 2,45,23,3

# Corpus stats
./etok stats --data corpus.txt

# Benchmark
./etok bench --data corpus.txt

# Morphology analysis
./etok rotate --a "walking" --b "walked"

---

Formats supported

Format	Example	Detection
Plain text	English, Spanish, any language	auto
CSV / TSV	a,b,c,d	auto
Code	func(); x=1;	auto
DNA / FASTA	ATCGATCG, >header\nATCG...	auto
RNA	AUCGAUCG	auto

No configuration needed. magic_split detects the format automatically.

---

DNA / FASTA support

etok detects DNA and FASTA files automatically:

# Works directly — no flags needed
./etok train --data MW182853.fasta --out sars_vocab.json --vocab_size 256 -v

Output on a 30KB FASTA file:

Mode      : DNA/FASTA (k-mer k=6)
Words     : 5000 total / 2890 unique (1.7x dedup)
Vocab: 256 | Merges: 247 | Train: 0.099s
Entropy: 2.31 -> 7.29

How it works:

1. FASTA header lines (>...) are stripped automatically
2. Sequence is joined and split into non-overlapping 6-mers: ATCGATCG → ATCGAT, CG...
3. BPE merges frequent k-mer pairs → learns codons, motifs, repeat units
4. The resulting vocabulary captures real biological patterns

Custom k-mer length:

./etok train --data genome.fasta --out vocab.json --kmer 3   # codons
./etok train --data genome.fasta --out vocab.json --kmer 8   # longer motifs

---

Benchmarks

Measured on 2-core dev machine (Intel, AVX2, 2 threads):

Tool	Train 168KB	Encode	Notes
etok v5	97ms	~600 MB/s	C
SentencePiece	~400ms	~170 MB/s	C++, external dep
HuggingFace tokenizers	~800ms	~100 MB/s	Rust, external dep
tiktoken	N/A	~1000 MB/s	Rust+PCRE2+JIT

etok on modern 8-16 core hardware (estimated):

• Encode: ~400–600 MB/s (OpenMP parallelizes over words, near-linear scaling)
• Train: still fastest (single-threaded training is not the bottleneck)

Why tiktoken is faster at encode: tiktoken uses PCRE2+JIT (a 600KB C library) compiled to native SIMD. etok is zero-dependency and within 2-3x of tiktoken on equivalent hardware.

DNA bench on 30KB FASTA (this machine):

vocab	train	encode	chars/tok
512	149ms	1.6ms	3.3
1024	246ms	1.6ms	3.8
2048	453ms	1.5ms	4.9
2910	603ms	1.5ms	5.7

---

Novel features (not in tiktoken / SentencePiece / HuggingFace)

magic_split — auto-detect separator

No configuration. Detects spaces, tabs, commas, semicolons, newlines, DNA format.

./etok train --data anything.txt --out v.json   # just works

freq × length — better merge ranking

All other BPE implementations rank merges by frequency only.
etok ranks by frequency × (len_a + len_b), which prefers longer tokens at equal frequency.
Result: ~5% better compression (fewer tokens per text = more effective context window).

entropy-stop — automatic vocab size

etok monitors token entropy during training and stops when adding more vocab produces diminishing returns. You can still specify --vocab_size as a cap.

rotate_compare — rotation-invariant morphology

Finds shared morphological roots between words using Z-function on rotated strings. Unique to etok. Useful for agglutinative languages (Turkish, Finnish) and circular DNA.

./etok rotate --a "walking" --b "walked"
# rotate_compare('walking','walked') = 'walk'

./etok rotate --a "ATCGATCG" --b "TCGATCGA"
# rotate_compare('ATCGATCG','TCGATCGA') = 'TCGATCG'  (circular DNA!)

---

Build options

# Full (AVX2 + OpenMP) — fastest
gcc -O3 -march=native -ffast-math -fopenmp -o etok etok.c -lm

# Portable (no SIMD, no OpenMP)
gcc -O3 -o etok etok.c -lm

# Static binary (deploy anywhere)
gcc -O3 -march=native -ffast-math -fopenmp -static -o etok etok.c -lm

Build works on Linux, macOS, Windows (MinGW). No cmake, no make, no dependencies.

---

vocab.json format

Human-readable JSON. Open in any text editor.

{
  "version": 5,
  "vocab_size": 512,
  "splitter": " ",
  "kmer_mode": 0,
  "kmer_k": 6,
  "vocab": {
    "<pad>": 0, "<unk>": 1, "<bos>": 2, "<eos>": 3,
    "the</w>": 5, "and</w>": 6, ...
  },
  "merges": [
    ["t", "h"], ["th", "e</w>"], ...
  ]
}

For DNA vocabs: kmer_mode: 1 is saved automatically and restored on load.

---

Architecture

Input text
    │
    ├─ magic_split()   detect format (text/CSV/code/DNA/FASTA)
    │
    ├─ [DNA mode]  extract_dna_seq() → k-mer split → BPE pool
    │   └─ strip FASTA headers, join sequence, split every k chars
    │
    ├─ [text mode] next_word() → SIMD presplit (AVX2 32 bytes/cycle)
    │
    ├─ build_pool()    word frequency table
    ├─ build_idx()     pair scores (freq × length) + inverted index
    ├─ max-heap BPE    O(n log n) merge loop
    ├─ entropy-stop    monitor bits/token, stop at diminishing returns
    └─ DAFSA build     compact trie (~40B/node vs 1024B dense trie)

Encode:
    SIMD presplit → OpenMP parallel DAFSA MaxMatch → concatenate
    O(n × log(avg_children))  ~4 children/node for BPE vocab

---

Comparison with other tokenizers

Feature	etok	tiktoken	SentencePiece	HuggingFace
Single file	✅	❌	❌	❌
Zero deps	✅	❌ (PCRE2)	❌	❌
DNA/FASTA	✅	❌	❌	❌
Auto separator	✅	❌	❌	❌
Readable vocab	✅ JSON	❌ binary	❌ protobuf	✅ JSON
freq×length rank	✅	❌	❌	❌
Entropy stop	✅	❌	❌	❌
Encode speed	~80–600 MB/s	~1000 MB/s	~170 MB/s	~100 MB/s
Train speed	✅ fastest	N/A	slower	slower

---

Donations

ETH - 0xD773B73C7ea4862020b7B5B58f31Ea491f5a9bA3

BTC - bc1ql6qvsk67hl5vz346kx4gueqjhp6me9ss8eflgt

SOLANA - GYBiTvVbPvPJP7ZK5oaqc9w6UtHvd6NkhSPP2UBhDvfh

License

Apache 2 - Non Comercial