PageIndex: Vectorless, Reasoning-Based RAG (github.com)

A new system called PageIndex has been introduced, revolutionizing the retrieval-augmented generation (RAG) approach by moving away from traditional vector databases and chunking. Instead, PageIndex relies on a hierarchical tree structure that models how human experts navigate complex documents. By employing reasoning-based retrieval, this system enhances document analysis for professionals dealing with lengthy texts, achieving a remarkable 98.7% accuracy on the FinanceBench benchmark, outperforming conventional vector-based methods that often prioritize similarity over relevance. The significance of PageIndex lies in its ability to simulate human-like reasoning and navigation through documents, offering better explainability and traceability. Users can generate a tree index from documents, allowing large language models (LLMs) to perform two-step retrieval based on this structure rather than vague vector comparisons. With PageIndex, professionals can expect a more accurate, context-aware analysis of intricate documents like financial reports or regulatory filings, making it a crucial tool for industries requiring detailed information extraction. The framework is open-source and can be run locally or integrated into existing systems via API, enabling widespread access to this innovative approach.