Debunking the Claims of K2-Think (www.sri.inf.ethz.ch)

K2-Think, a recently released reasoning-focused large language model (LLM), has garnered substantial media attention for claims of matching the performance of much larger models like GPT-OSS 120B and DeepSeek v3.1, despite using fewer parameters. However, a detailed analysis reveals these claims are overstated due to significant flaws in the model's evaluation methodology. Key issues include data contamination—where a substantial portion of the test problems were included in K2-Think’s training datasets—rendering the reported math and coding benchmark results invalid. Additionally, performance gains are amplified through unfair comparisons, such as using a best-of-3 scoring method assisted by an unspecified external model, while competitor models were evaluated with best-of-1 and without similar external help. Further discrepancies arise from misrepresenting results of competing models like GPT-OSS and Qwen3, often using outdated versions or suboptimal settings that artificially depress their scores. K2-Think also skews aggregate scores by weighting benchmarks dominated by contaminated data more heavily, misleadingly inflating overall performance metrics. Independent tests under fair conditions confirm K2-Think’s abilities fall well short of its stated claims, performing below comparable models of similar size. This case underscores the importance of rigorous, transparent evaluation practices in AI research to ensure credible benchmarking, especially as the community seeks trustworthy, efficient reasoning LLMs. While open models like K2-Think contribute valuable diversity, inflated claims based on flawed methodologies risk confusing progress and undermining trust in the field.