How fast is N tokens per second really? (mikeveerman.github.io)

A new tool has been introduced to help users better understand the speed metrics reported for Local Language Models (LLMs) by illustrating how many tokens per second (tok/s) really translate into visible output. Current benchmarks often showcase impressive numbers like "500 tok/s on Groq," but without context, these figures can be abstract. The tool features four distinct streaming modes—code, text, think, and agent—illustrating how output type influences perceived speed. For example, participants can observe the difference in flow rate between syntax-highlighted code and prose, highlighting the varying densities of tokens across different formats. This development is significant for the AI/ML community as it bridges the gap between raw throughput data and perceptual understanding. By aligning token speed with real-time outputs, users can better gauge the performance of LLMs in practical scenarios. Additionally, the tool uses a broad-based tokenization method, approximating BPE-style encoding rather than specific vendor implementations, which allows for a comprehensive understanding of how different input types affect processing rates. Overall, this initiative not only enhances transparency regarding LLM performance but also aids developers and researchers in optimizing model interaction and user experience.