AIs Favourite Number (sackfield.substack.com)

Recent observations reveal that when asked to generate a random number between 1 and 10, AI models, particularly large language models (LLMs), tend to cluster around the number 7. This tendency mirrors a common bias seen in humans, indicating that LLMs replicate human-like numeric biases shaped by the vast amounts of human-generated data they are trained on. This finding is significant for the AI/ML community as it highlights the potential pitfalls in using AI for evaluations that rely on numerical scoring, such as assessing code quality or rating products, without understanding the underlying distribution of these scores. Furthermore, leveraging LLMs as evaluators necessitates a shift from subjective continuous scoring to a more robust binary questioning approach. Rather than relying on a single score that could misrepresent performance due to the model's distribution biases, creating a series of binary questions allows for clearer, more objective assessments. This method not only diminishes the impact of rote biases but also provides a framework for generating more accurate continuous scores through weighted binary responses. Advanced techniques such as normalizing distributions via transformations (e.g., Yeo-Johnson) can further enhance the reliability of evaluations by ensuring smoother output gradients, allowing for better model optimization.