FrontierPhysics: Evaluating Agents for End-to-End Physics Research (arxiv.org)

FrontierPhysics has been introduced as a novel end-to-end benchmark designed to enhance the evaluation of AI agents in physics research. Unlike existing benchmarks that assess only isolated tasks or predefined solutions, FrontierPhysics encompasses 56 comprehensive tasks created by 42 physicists, spanning nine domains in theoretical, computational, and experimental physics. This benchmark requires AI agents to engage in the full research process, including literature review, research planning, experiment execution, and the generation of structured scientific manuscripts. It utilizes both deterministic verification methods and expert-authored rubrics to assess the correctness and publishability of the agents' research outputs. This advancement is significant for the AI/ML community as it highlights the need for more rigorous evaluation frameworks that go beyond basic correctness, enabling agents to emulate the actual research process. Initial tests of 22 different AI models under 26 configurations revealed that no configuration achieved more than 46 successes out of 168 attempts. Insights from these tests indicate that the primary challenges for these frontier agents lie in their reasoning skills and intuitive understanding of physics, rather than in their foundational knowledge or procedural execution. This underscores the potential for future developments to improve AI's role in scientific inquiry, bridging the gap between automation and human-like reasoning.