TaxCalcBench: Evaluating Frontier Models on the Tax Calculation Task (arxiv.org)

TaxCalcBench introduces a focused benchmark that tests whether frontier language models can compute U.S. personal income tax returns from scratch given all required inputs. The paper frames tax preparation as a combined language-understanding and precise numerical-reasoning task: models must parse dense legal prose, apply conditional rules, consult tax tables, and produce exact numeric results. The authors release data, code and demos and evaluate state-of-the-art models on a simplified sample of federal returns. Results are sobering: SOTA models correctly compute fewer than one-third of returns. Common failure modes include misusing tax tables, arithmetic and aggregation mistakes, and wrong eligibility or filing-status determinations. These systematic errors show that current LLMs struggle with deterministic, high-stakes rule execution even when the relevant facts are available. For the AI/ML community this underscores limits of end-to-end LLMs on tasks requiring exact symbolic computation and compliance with legal rules, and highlights the need for infrastructure—tooling for deterministic calculation, symbolic/constraint modules, verifiable computation, structured prompting and retrieval—to safely deploy models in regulated, safety-critical applications like tax filing.