AI Can't Read an Investor Deck (www.mercor.com)

Recent research by Anthropic highlights the limitations of frontier AI models—like GPT-5.4, Gemini 3.1 Pro, and Claude Opus 4.6—in handling complex financial documents, revealing significant gaps between their benchmark performance and real-world applicability. By testing these models on 25 tasks that required interpreting multimodal financials, including charts and graphs, the study found that models struggled to accurately read and reason from dense visual information. Across these tasks, performance plummeted when models were presented with image-only inputs; for instance, Claude Opus 4.6 and GPT-5.4 only achieved 4% accuracy based on visual extraction. This research is critical for the AI/ML community, particularly in the finance sector, as it underscores the need for more nuanced benchmarks that reflect the messy realities of financial analysis. Many standard assessments rely on simpler data formats, failing to challenge models with the complexities of real documents. The findings indicate that while there is ongoing progress in AI capabilities, a deeper understanding of their limitations is essential before concerns around AI displacement of financial roles escalate. Overall, this study calls for a reevaluation of AI’s readiness for sophisticated financial tasks amidst the industry's push for enhanced visual reasoning.