Production RAG: what I learned from processing 5M+ documents (blog.abdellatif.io)

After eight months building retrieval-augmented generation (RAG) pipelines for enterprise-scale corpora (examples: Usul AI with ~9M pages and a legal customer with ~4M pages), the team found that the YouTube-tutorial approach (LangChain → LlamaIndex) gets you a quick prototype but not production quality. They rewrote pieces iteratively and distilled what actually moved the needle, then released their learnings as an open-source project (agentset-ai/agentset, MIT). The big takeaway: prototyping hides many failure modes — production RAG needs deliberate engineering choices around query breadth, chunking, ranking and routing. Key technical takeaways: use LLM-driven query generation to expand a user’s context into multiple semantic + keyword queries and run them in parallel, feeding results into a reranker (their sweet spot: input ~50 chunks → output 15) — reranking often salvages suboptimal retrieval setups. Chunking deserves the most time: avoid mid-word/sentence splits and make each chunk a self-contained logical unit. Include relevant metadata (title, author) alongside chunk text to improve LLM responses. Add a lightweight query router to bypass full RAG for simple tasks (e.g., summaries, authorship). Stack notes: vector DBs evolved Azure → Pinecone → Turbopuffer; document extraction was custom; chunking used Unstructured.io plus bespoke flows; embeddings: text-embedding-large-3; rerankers: moved from none → Cohere 3.5 → Zerank; LLMs cycled GPT-4.1 ↔ GPT-5. These pragmatic choices and ROI-ranked optimizations are immediately applicable for teams moving RAG to production.