ComputeBench: Instruction-following benchmarks for long, step-by-step arithmetic (notdian.github.io)

The recent announcement of ComputeBench introduces a novel benchmarking tool specifically designed for evaluating models on long, step-by-step arithmetic tasks. This benchmark is significant for the AI/ML community as it highlights the performance of different models in understanding and executing complex arithmetic sequences, a capability critical for advancements in instructional-following AI applications. Notably, Google's Gemini-3 Pro Preview achieved a 100% match rate and answer accuracy across its trials, setting a new standard for performance in this domain. ComputeBench assesses models based on several metrics, including exact match rate, answer accuracy, and format adherence, helping researchers identify strengths and weaknesses in model outputs. For example, even prominent models like OpenAI's GPT-4.1 and Google's Gemini-2.5 Flash struggled, both recording zero exact matches, underscoring ongoing challenges in structured reasoning tasks. This benchmarking initiative underscores the importance of precise instruction-following capabilities in AI, with implications for improving algorithms that require detailed problem-solving strategies across multiple steps.