RAG Fails on PDF Tables (mrm1001.github.io)

Recent advances in Retrieval-Augmented Generation (RAG) pipelines highlight the critical role of effective OCR preprocessing, especially for extracting and embedding tables from academic PDFs. Tables and multimodal data are pervasive in scientific literature, yet current vision-language models (VLMs) still face significant challenges in accurately parsing complex document layouts, such as multi-column formats and embedded footnotes. This often leads to broken paragraphs, misplaced headers, and detachment of table captions from the data, undermining the reliability of retrieved answers. The study shows that including contextual information like table captions can improve answer accuracy by approximately 40%, emphasizing how vital fine-grained, context-aware document understanding is for trustworthy RAG systems. From a technical standpoint, the research evaluates various OCR approaches, comparing raw PDF text layers with markdown-converted tables processed by VLMs. Despite markdown offering a more machine-readable structure, VLMs sometimes misinterpret reading order or fail to link tables to their captions, which is straightforward in structured formats like XML. Using a benchmark dataset from the “OCR Hinders RAG” paper, the analysis reveals how OCR errors cascade and degrade the quality of RAG outputs, particularly on table-based questions. These insights stress the importance of advancing OCR techniques within foundational multimodal models to enable more precise extraction, verification, and retrieval of scientific knowledge—a step critical for applications like literature reviews and medical research, where accuracy and traceability are non-negotiable.