Agents' Last Exam (arxiv.org)

A new evaluation benchmark called Agents' Last Exam (ALE) has been introduced to address the significant gap between AI performance on standard benchmarks and its practical deployment in economically meaningful applications. Developed in collaboration with over 250 industry experts, ALE focuses on assessing AI agents on long-horizon tasks within non-physical industries, structured around a comprehensive taxonomy that includes over 1,000 tasks across 55 subfields and 13 industry clusters. This initiative underscores the need for sustained evaluation in real-world workflows, as current benchmarks have not successfully translated technical achievements into tangible economic benefits. The significance of ALE lies in its design as a dynamic benchmark that adapts to emerging workflows and industries, rather than functioning solely as a static leaderboard. Initial results indicate that the most challenging tasks are still far from being mastered, with an average pass rate of only 2.6%. By shifting the focus of AI evaluation towards economically impactful tasks, ALE aims to foster advancements that contribute more directly to economic growth and productivity, thereby bridging the existing evaluation gap in AI applications.