Asymmetric Quantization: Near-Lossless Retrieval with 97% Storage Reduction (www.mixedbread.com)

A recent announcement from Mixedbread Search details the implementation of asymmetric quantization in their late interaction retrieval system, Silo. This innovative optimization allows for significant storage reduction — achieving a 97% decrease in size while maintaining near-lossless retrieval quality. By storing document vectors as binary signs and keeping query vectors in higher precision (int8), the system manages to compress the average storage per document from 393 KiB to just 12.28 KiB, while experiencing only a slight drop in retrieval accuracy (89.65 NDCG@10 compared to 90.26 for full precision). This efficiency is crucial for scaling retrieval operations across billions of documents. The significance of this development lies in its promising implications for the AI/ML community, particularly in the realm of efficient data handling and model deployment. As late interaction models produce multiple embeddings for individual documents, the associated storage costs can escalate rapidly. Asymmetric quantization addresses these cost challenges by reducing the payload size without drastically impacting retrieval performance. This change not only optimizes resource usage but also enhances system speed, allowing for faster query responses and improved throughput. Consequently, it paves the way for broader adoption of advanced retrieval models in production environments, positioning Mixedbread Search as a leader in efficient AI implementations.