Skip to content

Tags: zerfoo/ztoken

Tags

v1.0.0

Toggle v1.0.0's commit message
chore(main): release 1.0.0

v0.3.4

Toggle v0.3.4's commit message
fix: use greedy longest-match for SentencePiece (matches llama.cpp)

Replace Viterbi DP with greedy leftmost-longest match in
sentencePieceEncode. This matches llama.cpp's llm_tokenizer_spm
behavior and fixes token splitting where sub-token score sums
beat whole-token scores (e.g., "▁What" splitting into "Wh"+"at").

v0.3.3

Toggle v0.3.3's commit message
fix: enable addLeadingSpace for SentencePiece unigram models

SetSentencePiece(true) now sets addLeadingSpace=true as a persistent
field on BPETokenizer, matching llama.cpp / SentencePiece default
behavior. Previously addLeadingSpace was only a parameter passed
through the call chain — making it a field ensures the first word
always gets the ▁ prefix prepended, so tokens like ▁What are found
by the Viterbi DP instead of falling back to character-level tokens.

Also adds SetAddLeadingSpace() for GGUF models that override the
default via tokenizer.ggml.add_space_prefix metadata.

v0.3.2

Toggle v0.3.2's commit message
fix: use large penalty for byte fallback in SentencePiece Viterbi

Byte fallback tokens (<0xNN>) were competing with multi-character vocab
tokens in the Viterbi DP using their actual vocabulary scores. When byte
token scores happened to be higher than vocab token scores, the Viterbi
algorithm preferred 43 byte-level tokens over 7 word-level tokens.

Fix: assign byte fallback tokens a fixed score of -1e6 instead of their
vocabulary score, ensuring they are only used as a last resort when no
vocab token covers a position. This matches llama.cpp behavior.

v0.3.1

Toggle v0.3.1's commit message
fix: implement Viterbi SentencePiece encoding (replaces greedy)

The greedy longest-match approach in sentencePieceEncode produced
suboptimal tokenization for SentencePiece unigram models (e.g.,
Mistral 7B). Replace it with Viterbi dynamic programming that finds
the segmentation maximizing the sum of log-probability scores.

Also adds:
- Byte fallback encoding/decoding via <0xNN> tokens for chars not in vocab
- decodeSentencePieceBytes for proper round-trip of byte fallback tokens
- Tests: Viterbi vs greedy, byte fallback, sentence round-trip, edge cases

v0.3.0

Toggle v0.3.0's commit message
feat: add SentencePiece unigram encoding for models without merges

SentencePiece unigram models (e.g., Mistral 7B GGUF) provide vocabulary
scores but no BPE merge table. Without this, encoding fails silently,
producing wrong token IDs and garbage output.

Add SetScores() to BPETokenizer and a greedy longest-match encoder that
selects tokens by length first, then by score. When merges are empty but
scores are present, encodeSegment automatically uses this path instead
of BPE merging.

Also extend the gguf.Metadata interface with GetFloat32Array and extract
tokenizer.ggml.scores in ExtractTokenizer so GGUF-loaded tokenizers
automatically use unigram encoding when appropriate.

v0.2.0

Toggle v0.2.0's commit message
chore(release): prepare v0.2.0

v0.1.0

Toggle v0.1.0's commit message
docs: align README with project conventions