MathDuels: Evaluating LLMs as Problem Posers and Solvers (arxiv.org)

A new benchmark called MathDuels has been introduced to more effectively evaluate the capabilities of frontier language models (LLMs) in both problem creation and problem-solving. Unlike traditional assessments that only position models as solvers of static mathematical challenges, MathDuels allows models to take on dual roles: they generate math problems through an innovative three-stage pipeline and solve problems authored by other models. This dynamic approach reveals a more nuanced understanding of model strengths, exposing capabilities that single-role benchmarks may overlook. The significance of MathDuels lies in its ability to foster continuous improvement in model performance. As new models emerge, they generate increasingly challenging problems, ensuring that the benchmark evolves and reflects the current state of AI capabilities. By employing a Rasch model to assess solver abilities and problem difficulties, this method provides a sophisticated framework for understanding how well models can create and solve problems. The establishment of a public leaderboard adds a competitive element, demonstrating the benchmark's potential to drive advancements in AI mathematics and machine learning methodologies.