A benchmark of expert-level academic questions to assess AI capabilities – HLE (www.nature.com)

A new benchmark called Humanity’s Last Exam (HLE) has been introduced to better assess the capabilities of large language models (LLMs), which have demonstrated high accuracy on existing benchmarks but struggle with more complex academic questions. HLE features 2,500 challenging questions spanning diverse fields such as mathematics, humanities, and natural sciences, created by subject-matter experts from over 500 institutions globally. This benchmark presents questions that are not easily answerable through simple internet searches, emphasizing expert-level knowledge and reasoning skills. The significance of HLE lies in its design as a rigorous evaluation tool that highlights the current limitations of LLMs, which exhibit low accuracy and poor calibration on these expert-level tasks. The introduction of HLE aims to provide a precise measure of AI capabilities against the expert human frontier, encouraging advancements in LLM performance. By publicly releasing this benchmark, the developers seek to catalyze further research and discussions surrounding model proficiency and its implications for AI development, transparency, and governance. The dataset is expected to inspire a competitive environment among researchers, backed by a substantial prize pool for high-quality question submissions and a structured review process to ensure question integrity.