Benchmarks in Leipzig (arxiv.org)

In a groundbreaking workshop called "Benchmarks in Leipzig," a team of 49 mathematicians gathered at the Max Planck Institute for Mathematics in Leipzig to compile a dataset of 100 research-level math questions, all of which have known answers. This extensive project involved evaluating several state-of-the-art large language models (LLMs) over three stages. Initially, five models tackled the questions, leaving 41 unanswered. After further evaluations with three models, this number dropped to 16, and following the final assessments with two advanced models, only two questions remained unsolved. This initiative is significant for the AI/ML community as it highlights the substantial advancements in the mathematical reasoning abilities of LLMs. The progressive improvement in problem-solving capabilities across the evaluation stages illustrates the potential for these models to tackle increasingly complex mathematical challenges. The findings also provide a valuable benchmark for future AI applications in mathematics, offering insights into the strengths and limitations of current LLMs, ultimately guiding the development of even more sophisticated systems.