Tags: zerfoo/ztoken
Tags
fix: use greedy longest-match for SentencePiece (matches llama.cpp) Replace Viterbi DP with greedy leftmost-longest match in sentencePieceEncode. This matches llama.cpp's llm_tokenizer_spm behavior and fixes token splitting where sub-token score sums beat whole-token scores (e.g., "▁What" splitting into "Wh"+"at").
fix: enable addLeadingSpace for SentencePiece unigram models SetSentencePiece(true) now sets addLeadingSpace=true as a persistent field on BPETokenizer, matching llama.cpp / SentencePiece default behavior. Previously addLeadingSpace was only a parameter passed through the call chain — making it a field ensures the first word always gets the ▁ prefix prepended, so tokens like ▁What are found by the Viterbi DP instead of falling back to character-level tokens. Also adds SetAddLeadingSpace() for GGUF models that override the default via tokenizer.ggml.add_space_prefix metadata.
fix: use large penalty for byte fallback in SentencePiece Viterbi Byte fallback tokens (<0xNN>) were competing with multi-character vocab tokens in the Viterbi DP using their actual vocabulary scores. When byte token scores happened to be higher than vocab token scores, the Viterbi algorithm preferred 43 byte-level tokens over 7 word-level tokens. Fix: assign byte fallback tokens a fixed score of -1e6 instead of their vocabulary score, ensuring they are only used as a last resort when no vocab token covers a position. This matches llama.cpp behavior.
fix: implement Viterbi SentencePiece encoding (replaces greedy) The greedy longest-match approach in sentencePieceEncode produced suboptimal tokenization for SentencePiece unigram models (e.g., Mistral 7B). Replace it with Viterbi dynamic programming that finds the segmentation maximizing the sum of log-probability scores. Also adds: - Byte fallback encoding/decoding via <0xNN> tokens for chars not in vocab - decodeSentencePieceBytes for proper round-trip of byte fallback tokens - Tests: Viterbi vs greedy, byte fallback, sentence round-trip, edge cases
feat: add SentencePiece unigram encoding for models without merges SentencePiece unigram models (e.g., Mistral 7B GGUF) provide vocabulary scores but no BPE merge table. Without this, encoding fails silently, producing wrong token IDs and garbage output. Add SetScores() to BPETokenizer and a greedy longest-match encoder that selects tokens by length first, then by score. When merges are empty but scores are present, encodeSegment automatically uses this path instead of BPE merging. Also extend the gguf.Metadata interface with GetFloat32Array and extract tokenizer.ggml.scores in ExtractTokenizer so GGUF-loaded tokenizers automatically use unigram encoding when appropriate.