Human Typing Habits and Token Counts (pankajpipada.com)

A recent exploration of human typing habits reveals how common errors and shorthand can significantly impact token counts when using AI models that bill per token. It highlights that typical typing tendencies, such as typos or the use of filler words, often lead to inflated token counts without altering the intended message. For instance, a simple prompt resulted in numerous tokens due to just a few spelling mistakes, illustrating the discrepancies between different AI systems like OpenAI and Claude's tokenization methods. This finding is crucial for the AI/ML community as it underscores the need for better awareness of how human typing patterns interact with AI billing structures. The technical implications are significant—small variations in text can drastically affect costs, especially in coding and communications that exhibit frequent typographic errors or use of specific word forms. As developers and users rely on these models for efficiency, understanding the nuances in tokenization can lead to more mindful interactions with AI systems, ultimately optimizing both cost and output quality.