How we engineered RAG to be 50% faster (elevenlabs.io)

The team behind a Retrieval-Augmented Generation (RAG) system has engineered a major speedup by redesigning its query rewriting step, cutting latency by 50%. RAG improves AI accuracy by embedding user queries, retrieving relevant context from large knowledge bases, and feeding that context into language models. However, query rewriting—collapsing dialogue history into precise, standalone queries to improve retrieval relevance—had become a bottleneck, accounting for over 80% of RAG’s latency due to reliance on a single, externally-hosted LLM. To address this, they implemented a “model racing” approach where the rewriting step runs multiple models in parallel, including self-hosted Qwen models, with the fastest valid response winning. This parallelism halved the median latency from 326ms to 155ms, drastically improving responsiveness without sacrificing accuracy. The system also falls back gracefully to raw user queries if no model responds quickly, ensuring fluid conversation even during peak loads or outages of external providers. The architecture’s robust internal model hosting smooths performance variability and enhances system resilience. This advancement is significant for AI/ML applications relying on RAG, especially conversational agents operating over large enterprise knowledge bases. By slashing query rewriting overhead to under 200ms, it removes a critical latency barrier, enabling real-time, context-aware interactions at scale without performance trade-offs—a key step toward more responsive and reliable AI-driven dialogue systems.