We gave nine LLMs the same PDF. Two answered without reading it (theaijudge.com)

A recent exploration involving nine large language models (LLMs) revealed significant disparities in their capabilities to read and respond to content from PDF documents. When researchers posed the same question related to a synthetic 27-page tender document, two models provided plausible answers without actually accessing the document, raising concerns about reliability in AI systems. The testing identified clear distinctions in LLM behavior: while some models effectively read the PDF and answered correctly, others either failed completely or generated inventive responses based on general knowledge instead of the document's content. This finding is crucial for the AI/ML community, particularly for developers integrating document processing capabilities into applications. It underscores the importance of not relying solely on status codes (such as a 200 OK response) to confirm that a model has engaged with a provided file. The research emphasizes the need for systematic verification, suggesting that applications should implement checks to ensure models have accurately processed documents, thereby improving the reliability of AI-driven responses in practical scenarios. Key technical insights included limitations on file size, the potential for hidden errors in preliminary capability descriptions, and variations in cost per model for document analysis.