Dayjob: Tax – evaluating AI agents on 50 professional tax assignments (surgehq.ai)

Surge AI has launched DAYJOB: Tax, a new benchmark evaluating AI agents on complex tax assignments that require specialized knowledge and professional judgment. The assessment consists of 50 scenarios across multiple areas of tax practice, challenging AI to produce accurate tax returns and advice, while recognizing incomplete or conflicting evidence. A recent test highlighted these challenges; although one AI agent correctly identified potential tax benefits for a homeowner selling a condo, it failed to inform her of a critical deadline, demonstrating limitations in recognizing consequential client issues. This benchmark is significant for the AI/ML community as it emphasizes the need for AI systems to not only process information but also exercise professional judgment akin to human experts. The results revealed that the leading AI models achieved a pass rate of only around 10%, underscoring the ongoing difficulty of programming AI to navigate nuanced professional tasks. As such, DAYJOB: Tax represents a crucial step in assessing and advancing the capabilities of AI agents in professional settings, revealing both their potential and the gaps that still exist in their ability to handle real-world complexities.