Token Visualizer (github.com)

Token Visualizer is a lightweight tokenizer playground that lets you inspect how different Hugging Face tokenizers split and encode a prompt. A model dropdown includes many popular tokenizers (GPT-2/Neo/OPT, LLaMA test tokenizer, Mistral, BERT, RoBERTa, T5, etc.), and the UI shows a color-coded token stream with token text, numeric IDs and highlighted special tokens. You can copy the token list to the clipboard for sharing or debugging. Public tokenizers (e.g., gpt2, bert-base-uncased) run without credentials; gated models (for example a real LLaMA tokenizer) require you to paste a Hugging Face access token which is used in requests and kept only in memory. This tool is significant for prompt engineers, model integrators and researchers because tokenization differences drive model behavior, cost and truncation risk. It makes subword boundaries, special-token placement and token IDs explicit, helping you estimate token counts, budget context windows, diagnose clipping/truncation, and compare BPE vs. WordPiece-style splits across architectures. The in-memory HF token flow lets you test private tokenizers without persistent exposure. Overall, Token Visualizer is a practical debugging and educational aid for anyone working on prompt design, token-budgeting or tokenizer compatibility between models.