90% of US working hours are spent on tasks not covered well by AI benchmarks (www.mercor.com)

A recent analysis reveals that 90% of working hours in the U.S. are dedicated to tasks inadequately captured by existing AI benchmarks, comprising a significant gap in AI's ability to address economically valuable work. This study introduced a "Benchmark Map," which highlights that only 208 out of 923 U.S. occupations have tasks associated with a minimum of five benchmark prompts. The findings indicate a skew in benchmark coverage, as tasks in high-wage, AI-exposed professions like law are well-represented, while essential roles in smaller, middle-income sectors, such as legal secretaries, remain overlooked. This gap is crucial for the AI/ML community as it highlights the challenges in assessing AI model capabilities in real-world job contexts. Understanding the coverage of existing benchmarks is vital for organizations aiming to identify suitable AI models and for policymakers evaluating AI's economic implications. As the authors work to expand the benchmark list and improve the representation of diverse jobs, they aim to create a more comprehensive overview of AI capabilities that reflects the needs of the entire labor market, moving beyond the heavily saturated sectors currently prioritized.