SHIFTing Languages in Multilingual RAG (qdrant.tech)

A new approach in multilingual retrieval-augmented generation (RAG) called SHIFT (Semantic Harmonization via Index-side Feature Transformation) promises significant improvements in cross-language information retrieval without the hefty costs of traditional methods. Announced by Evgeniya Sukhodolskaya, SHIFT addresses the prevalent issue of language bias in knowledge retrieval systems, wherein users querying in one language often receive answers only from that language, failing to access potentially more relevant responses in others. This phenomenon is particularly problematic for EU companies that operate with multilingual documentation. The SHIFT technique models each language as a consistent offset in vector space, enhancing the retrieval of semantically similar documents across different languages. By calculating offsets from translation pairs, documents are adjusted during indexing to account for these offsets, allowing queries in one language to find relevant content in another. Testing with the multilingual-e5-large embedding model showed an increase in cross-language retrieval metrics—from 0.633 to 0.737 in average nDCG@20—while the method maintains compatibility with various multilingual models. This innovation could reduce reliance on costly translation pipelines, making it a game-changer for organizations grappling with multilingual data. However, users must weigh potential trade-offs in same-language recall to ensure their specific needs are met.