fix(tokenizer): consume upstream tokenizers, retire forked impl (#52)#124

Merged

michalharakal merged 3 commits into

fix/52-tokenizer-upstream-gguf

May 4, 2026

Commits on May 4, 2026

fix(llm-core): route GGUF tokenizer load through upstream sk.ainet.io.tokenizer (#52)

michalharakal
and
claude
committed
feat(llm-core): add SentencePieceSpecialTokens decorator for Gemma-style chat models

michalharakal
and
claude
committed
feat(llm-core): wire SentencePiece decorator + add fromGgufSource / fromTokenizerJsonString

michalharakal
and
claude
committed