ArXivMath and BrokenArXiv: Harder Problems and Revised Grading (matharena.ai)

Recent updates to the ArXivMath and BrokenArXiv benchmarks have been prompted by advancements in AI models, particularly GPT-6 Astra, which achieved scores of 94% and 96% on these test sets, respectively. The benchmarks have been revised to include more challenging problems focused on recently refuted conjectures, utilizing coding tools like Python and SageMath. Notably, these changes allow models to be run within their optimized environments, ensuring a fairer assessment of their capabilities. Despite these heightened difficulties, GPT-6 Astra maintained its lead, scoring 88% on ArXivMath and 81% on BrokenArXiv, with updates to the grading system to better reflect nuanced understanding, such as identifying false claims. The significance of these adjustments lies in their potential to extend the relevance of the benchmarks by ensuring they evolve alongside AI advancements. The heightened problem complexity aims to filter out overconfident responses that don't accurately represent a model’s capabilities, while the introduction of a new grading mechanism facilitates a more rigorous evaluation. This evolution in testing methodology highlights the ongoing arms race in AI development, as the benchmarks must now continuously adapt to keep pace with emerging models. Additionally, attempts to expand the benchmarks into theoretical physics and computer science were largely unsuccessful, as GPT-6 Astra quickly saturated those domains as well, leading to a pivot back to focusing on mathematics.