670 MB/s Swift tokenizer inspired by Marcel Roed’s gigatoken. Zero-copy, NEON/CRC32, WebAssembly, and Embedded Swift.
swift high-performance neon tokenizer zero-copy swift-package-manager webassembly wasm simd arm64 tokenization core-ai bpe byte-pair-encoding large-language-models llm tiktoken mlx-swift embedded-swift gigatoken
-
Updated
Jul 31, 2026 - Swift