DiskBBQ – Elasticsearch's vector storage format (www.elastic.co)

DiskBBQ, a new vector storage format for Elasticsearch, presents an innovative alternative to the widely praised Hierarchical Navigable Small Worlds (HNSW) algorithm. By utilizing a Hierarchical K-means approach, DiskBBQ partitions vectors into smaller clusters, efficiently handling lower-memory scenarios while maintaining strong query performance. This design allows for vector representation to be queried in a multi-layered fashion, improving efficiency and minimizing memory issues that plague HNSW when working with large datasets. Moreover, DiskBBQ leverages Better Binary Quantization (BBQ) to optimize vector storage and access, reducing memory requirements while enabling faster bulk scoring operations. The significance of DiskBBQ lies in its ability to outperform HNSW in low-memory situations, where the latter's performance degrades significantly. Testing results show that while HNSW excels at high recall and low latency with ample memory, DiskBBQ offers impressive indexing speeds—up to 10 times faster than HNSW when the index fits entirely in RAM, and it maintains effective performance as memory decreases. For users seeking a cost-effective solution that balances speed and recall (around 95% or less), DiskBBQ opens new avenues for applications in AI and machine learning, especially in resource-constrained environments. The format is currently in technical preview within Elasticsearch Serverless, inviting developers to explore its capabilities firsthand.