Hybrid GPU–CPU Approach to Faster Vector Indexing and Cheaper Queries (milvus.io)

NVIDIA has introduced a hybrid GPU–CPU approach to enhance the performance of vector indexing and reduce query costs in large-scale AI applications, utilizing the CAGRA algorithm in Milvus 2.6.1. As vector databases are increasingly required to handle billions of embeddings, traditional CPU-based approaches struggle with efficiency during index construction. CAGRA leverages GPU capabilities for constructing high-quality neighborhood graphs through parallel processing, significantly speeding up the build time. However, querying these graphs on GPUs can be cost-prohibitive and less scalable. The new design alleviates this by using GPUs solely for index building while executing queries on CPUs, balancing the performance benefits of GPU speed with the cost-effectiveness of CPU deployments. This approach not only retains the advantages of the GPU-built indices, ensuring high recall and low-latency query performance, but also addresses the operational challenges of resource utilization and scalability. The flexibility provided by the adapt_for_cpu parameter allows Milvus to dynamically switch between GPU and CPU modes based on specific workload demands. Initial tests indicate that GPU CAGRA builds indexes 12–15 times faster than the traditional HNSW algorithm, and queries on GPU deliver 5–6 times higher throughput than CPU queries. This hybrid system thus opens new avenues for deploying efficient AI solutions in cost-sensitive, high-query-volume environments.