RAG retrieval accuracy depends on chunking. I measured how much (medium.com)

A recent experiment demonstrated that the accuracy of Retrieval-Augmented Generation (RAG) relies heavily on how documents are chunked. By segmenting the same two documents using four distinct chunking methods, the study revealed that a naive character limit approach yielded only 50% accuracy for a one-page table and a dismal 25% for a three-page table. In contrast, using layout-aware chunking improved the accuracy to 96% for the one-page table, while a final refinement—stitching together multi-page tables—boosted the three-page table's accuracy to 92%. These findings highlight the critical importance of chunking strategies in optimizing RAG performance. This research is significant for the AI/ML community as it underscores that poor chunking can often lead to inadequate model responses, rather than flaws in the model or prompts themselves. The study advocates for chunking that aligns with a document's structural elements, such as repeating headers in table chunks, and emphasizes the need for robust parsing mechanisms to maintain context. By providing detailed benchmarks and methods for replication, the findings invite practitioners to reassess their chunking strategies to enhance the effectiveness of their retrieval systems.