PageIndex (19k stars) scored 44% on legal docs. Same as vector RAG (medium.com)

Last week, testing revealed that the PageIndex architecture, despite its impressive claim of 98.7% accuracy on FinanceBench, struggled significantly when retrieving exact legal text from the EU General Data Protection Regulation (GDPR). In a series of tests against specific legal queries, PageIndex achieved only 44% needle coverage, similar to other retrieval-augmented generation (RAG) models. While PageIndex performed well in directing users to the correct sections of the document, it returned summaries rather than verbatim text, a critical factor for legal contexts where precise language matters. This raises concerns about the risk of hallucinations in summarization, underlining the necessity for exact text retrieval in legal compliance. The findings indicate a broader implication for the AI/ML community: the architecture of retrieval systems influences performance significantly across different document types. The experiment showed that changes in architecture shift failure modes rather than eliminate them. RAG systems must be aware of the distinctions between binding legal language and explanatory context to improve retrieval accuracy. As solutions like RagTune are developed to measure retrieval performance on specific documents, it emphasizes the importance of tailored benchmarks for different types of materials, suggesting that robust results in one domain do not guarantee success in another.