Production RAG: what I learned from processing 5M+ documents (blog.abdellatif.io)

After eight months building production RAG for large customers (including projects spanning millions of pages), the author reports that prototyping with LangChain + LlamaIndex is fast but deceptive: end-user quality often lags unless you invest in several engineering hotspots. The single biggest gains came from automated query generation (having an LLM produce many semantic + keyword queries from a thread, run in parallel, then rerank results) and a solid reranker—tiny code changes but outsized ROI. They found an effective reranker workflow was feeding ~50 candidate chunks and returning the top ~15. Practical, high-impact details: chunking is the hardest, most time-consuming piece—ensure chunks are logical, not cut mid-sentence, and capture standalone context. Inject chunk metadata (title, author, etc.) into the LLM alongside text for much better answers. Add a lightweight query router to handle non-RAG intents (e.g., "who wrote this") with direct API+LLM responses. Their stack evolved (Azure → Pinecone → Turbopuffer for vectors; custom extraction; Unstructured.io + custom chunkers; OpenAI text-embedding-large-3; reranker progression None → Cohere 3.5 → Zerank; LLMs GPT-4.1 ↔ GPT-5). All learnings are open-sourced at agentset-ai/agentset (MIT). Bottom line: production RAG is more engineering than plug‑and‑play—focus on query expansion, reranking, chunking, and metadata to move the needle.