Overview of the DiskANN Project (2018–present) (harsha-simhadri.org)

The DiskANN project, initiated in 2018, tackles the growing demands of large-scale vector search by significantly advancing state-of-the-art algorithms to better align with industrial requirements. Developed through collaboration among top institutions and Microsoft Research, DiskANN addresses major scalability and efficiency challenges in approximate nearest neighbor (ANN) search, resulting in innovations that improve performance by an order of magnitude. The project has not only generated influential research but also produced an open-source implementation widely adopted both within Microsoft and across the AI/ML industry, inspiring new hardware adaptations and fostering community benchmarking efforts. Recognizing limitations in earlier versions—particularly DiskANN’s tight coupling to specific storage hierarchies and difficulty integrating with databases—the team has since rewritten the system in Rust. This redesign aims to modularize DiskANN as a pluggable vector search engine compatible with diverse databases and memory tiers, offering flexibility in cost-performance optimization. The revamped DiskANN supports multiple backends and can operate alongside or in competition with prominent vector libraries such as FAISS and hnswlib. Its integration with Azure Cosmos DB, a geo-distributed NoSQL database, marks a milestone by embedding robust vector indexing directly into operational databases, making it competitive with dedicated serverless vector search systems. This evolution positions DiskANN as a practical, scalable foundation for embedding vector search in real-world AI applications.