Pure Elixir TikToken-style byte-level BPE tokenizer (Kimi K2 compatible).
nlp unicode encoding erlang elixir otp beam regex tokenizer hex-package tokenization kimi bpe byte-pair-encoding huggingface typespecs llm tiktoken moonshotai nshkr-ai-infra
-
Updated
Dec 29, 2025 - Elixir