Do we still need OCR? An implementation of a pure vision-based agent (pageindex.ai)

A new blog argues that traditional OCR — the two-stage layout detection plus text recognition used to feed LLMs — is fundamentally lossy and therefore limits document question-answering (QA). From an information‑theoretic view the OCR step is non-invertible and can discard layout, spatial, and visual cues that downstream reasoning needs. The authors demonstrate an alternative: end-to-end vision-language model (VLM) pipelines (examples: Qwen‑VL, GPT‑4.1, DeepSeek‑OCR) that operate directly on page images, preserving multimodal signals and enabling more faithful QA without an intermediate 1D text representation. Code and a notebook for the demo are available on GitHub. The post also tackles scaling: VLMs face context-window limits and "context rot" on long documents, while embedding-based retrievals (text or image) often fail on repetitive, domain-specific content. Their solution, PageIndex, generates an LLM-friendly table-of-contents as an in‑context index that guides page selection; selected page images are then sent to the VLM for detailed reasoning. This vectorless, retrieval-first design keeps spatial layout and visual semantics intact and mimics how humans use a ToC. The authors concede OCR still has value for pure 1D text tasks or as an auxiliary regularizer during training, but argue that VLM + PageIndex is a strong, information‑preserving alternative for complex, layout-rich documents.