DS-Serve: A framework for efficient, scalable neural retrieval (berkeley-large-rag.github.io)

Researchers from the University of California, Berkeley, the University of Illinois Urbana–Champaign, and the University of Washington have announced the launch of *DS Serve*, a comprehensive framework designed for efficient and scalable neural retrieval. This innovative system allows users to transform large in-house datasets—up to one trillion tokens—into a high-throughput retrieval service with a web UI and API. *DS Serve* boasts capabilities of up to 10,000 queries per second while using less than 200 GB of RAM, making it significantly more efficient than current commercial search engines. This prototype, which incorporates a massive 400B-token dataset, delivers search performance on par with established commercial endpoints without the associated costs and limitations. The significance of *DS Serve* lies in its ability to overcome longstanding challenges in neural retrieval, such as achieving high throughput and maintaining accuracy across extensive datasets. It combines advanced algorithms like DiskANN, which optimizes storage by using compressed vectors in RAM while maintaining full-precision vectors on SSDs. This results in superior performance compared to traditional methods like IVFPQ, addressing the trade-offs between accuracy, latency, and memory usage. *DS Serve* opens avenues for applications such as Retrieval-Augmented Generation (RAG), data attribution, and efficient training of search agents, thereby offering a powerful tool for the AI/ML community that seeks to leverage large-scale data effectively.