Dense Retrievers Know More Than They Can Express (www.mixedbread.com)

A recent exploration into dense retrieval models reveals that they possess richer representations than previously thought, raising important questions about their expressiveness amidst scoring limitations. While traditional single-vector models are constrained by their inability to capture complex information effectively, multi-vector models leveraging scoring operators like MaxSim demonstrate significantly stronger performance. This shift underscores the importance of scoring mechanisms in how retrieval models learn to represent information, emphasizing that models may inherently contain more knowledge than they can express due to the confines of their operators. Moreover, the introduction of Sparse AutoEncoders (SAEs) offers a promising avenue for extracting these latent representations without additional training. By imposing a sparsity constraint during encoding, SAEs can map features to a latent vocabulary that mirrors natural language distributions, adhering to Zipf's Law. This results in a vocabulary that is both interpretable and compatible with traditional retrieval methods, such as BM25. The latent features extracted reveal distinct categories that align well with semantic concepts, indicating that further integration of these findings could enhance the capability of retrieval systems in handling complex queries effectively. This research not only highlights the hidden potential within dense retrieval models but also paves the way for advancing interpretability and efficiency in AI retrieval technologies.