The Embedding Dilemma: Why Your RAG Fails and How to Think in Chunks (rewire.it)

The article argues that monolithic document embeddings—the single-vector “summary” of a long text—are fundamentally broken for modern Retrieval-Augmented Generation (RAG). A single averaged vector drowns out specific facts (the “Curse of the Average Vector”) and often only represents the first chunk of a document because of context-window limits. The practical fix is chunking: split documents into semantically coherent pieces and index those vectors. Approaches range from naive fixed-size splits (fast but error-prone), to recursive character splitting (robust baseline that respects paragraph/sentence boundaries), to semantic chunking (SOTA: embed candidate pieces and detect topic-boundary drops to form coherent chunks). Chunk size is a critical hyperparameter—Bhat et al. (2025) found 64–128 tokens work best for fact Q&A while 512–1024 tokens suit narrative summaries—so tune to your task. To balance precision and context, researchers propose “situated embeddings” (e.g., SitEmb): embed a small retrieval-focused chunk while showing the model surrounding context so the vector is precise yet context-aware. At scale, flat indexes explode, so use hierarchical indexing (document summaries → section summaries → granular chunks) to prune search top-down efficiently. Bottom line: stop using monolithic embeddings for RAG, start with a recursive splitter, treat chunk size as a hyperparameter, and explore situated embeddings plus hierarchical indexes for performant, scalable retrieval.