ANN v3: 200ms p99 query latency over 100B vectors (turbopuffer.com)

The recent announcement of Approximate Nearest Neighbor (ANN) Search v3 marks a significant advancement in vector search technology, allowing for efficient querying of over 100 billion vectors with a p99 latency of just 200 milliseconds. Developed within the turbopuffer architecture, this system amalgamates cutting-edge hardware optimization strategies with novel algorithms, aiming to facilitate high query rates and scalability, making it a game-changer for the AI/ML community. To achieve this performance, ANN v3 leverages two primary techniques: hierarchical clustering and binary quantization. Hierarchical clustering reduces the search space by organizing vectors into a multi-dimensional tree structure, optimizing memory access and minimizing bottlenecks. Simultaneously, binary quantization compresses the vector sizes by as much as 32 times, allowing for improved memory hierarchy utilization and bandwidth, thereby enhancing throughput. Together, these innovations facilitate faster and more efficient vector searches, essential for applications in machine learning and large-scale data analysis.