GSM8K problems can't tell Haiku 4.5 from Opus 5 (github.com)

A recent analysis comparing Claude models Haiku 4.5 and Opus 5 on the GSM8K benchmark revealed that the two models produced nearly identical results across 100 problems, with only a slight differentiation found in 2% of the cases. This examination utilized a method without an LLM judge to avoid biases from grading noise, resulting in both models achieving high pass rates (98.0% for Haiku 4.5 and 98.7% for Opus 5). Ultimately, the experiment cost just $1.68, underscoring the efficiency of this testing approach. The significance of this finding lies in the realization that the GSM8K benchmark may not effectively distinguish between models operating at high performance levels, as both fell near the benchmark's ceiling. While the analysis indicates both models are capable, it highlights the challenge of interpreting performance metrics when models perform similarly well. The results emphasize that although the benchmark can detect regression, it lacks the capacity to rank models when they exceed a certain threshold, steering the AI/ML community toward a more nuanced understanding of model comparison metrics in saturated testing environments.