Can AI agents do senior expert work? What the evidence says (estimandlab.com)

Recent studies have assessed the capability of AI agents to perform senior expert tasks, raising questions about their reliability in producing expert-level work. Findings indicate that while AI can excel in specific contexts, such as completing tasks more quickly and efficiently, it often falters in open-ended judgment and method selection. For instance, a study using GDPval kicked off a rigorous comparison of AI outputs against human work across various professions, demonstrating that AI can produce valuable insights, albeit in a limited capacity. One significant takeaway is that AI's effectiveness varies greatly depending on the task's complexity and the surrounding context. Moreover, another study involving consultants at Boston Consulting Group revealed that while AI-assisted teams processed tasks faster and with improved quality within defined technological limits, their effectiveness drastically decreased in more complex situations, showing a 19-point performance drop compared to their non-AI counterparts. These findings highlight a paradox: AI can enhance productivity on simpler tasks but may introduce errors in more nuanced scenarios. As reliance on AI for critical analysis grows, the community is urged to rigorously evaluate AI outputs against expert benchmarks, particularly in challenging cases, and to continuously assess the tools' real-world efficacy rather than relying solely on subjective measures of "feeling" productive.