Tokenizer playground to see how different Hugging Face tokenizers split your prompt.
- Model dropdown with popular tokenizers (GPT-2/Neo/OPT, LLaMA test tokenizer, Mistral, BERT, RoBERTa, T5, etc.).
- Color-coded token stream with IDs and special-token highlighting.
- Clipboard copy for the token list.
- Optional Hugging Face access token for gated models (kept in-memory only).
- Public tokenizers (e.g.,
gpt2,bert-base-uncased) work without a token. - Gated models (e.g., real LLaMA) need an HF access token—paste it in the field and requests will include it. The token stays in memory only.
Support the repository by joining the stargazers for this repo ⭐
If you found this project helpful or you learned something from the tutorials and want to thank me, consider buying me a cup of ☕
