LLMs can't read PDFs in 2026? (musings-mr.net)

In a recent evaluation of large language models (LLMs) for parsing complex PDF documents, results reveal significant shortcomings in their ability to accurately read municipal finance reports. Despite advancements in AI, models like Gemini, Claude, and ChatGPT struggled to achieve over 70% accuracy in recognizing essential data, with specific recalls as low as 31.1% in some cases. This inadequacy underscores the challenges LLMs face in document understanding, particularly with structured financial information presented in varying formats, where models often fail to recognize vital components or make costly errors in data extraction. This situation is noteworthy for the AI/ML community as it highlights the transition towards specialized models for document processing over general-purpose LLMs. Research indicates that smaller, document-focused vision models (VLMs) have outperformed state-of-the-art LLMs in parsing accuracy. The emergence of a "decoupled-VLM" architecture, emphasizing layout detection followed by specific region recognition, proves more effective for complex tables commonly found in financial documents. This shift could pave the way for more reliable and efficient AI tools in extracting structured information from PDFs, crucial for applications in finance, governance, and enterprise data management.