Multimodal embeddings outperform text on visual docs but lose on pure text (agentset.ai)

Recent research has revealed that multimodal embeddings outperform traditional text embeddings when dealing with visual documents, such as tables and charts, while the reverse holds true for pure text documents. A controlled study evaluated two retrieval approaches across datasets including DocVQA and ChartQA, comparing text-based retrieval using OpenAI's embedding model against a native multimodal retrieval method employing Voyage Multimodal 3.5. The findings showed that while text embeddings excelled in Recall@1 performance for pure text documents (96% vs. 92%), multimodal embeddings significantly outperformed in retrieving tables (88% vs. 76%) and slightly edged ahead with charts (92% vs. 90%). The significance of this study lies in its implications for optimizing information retrieval strategies across varied document types. For purely textual content, the established text embedding methods suffice, but for documents with visual components, especially those with complex layouts like tables, multimodal embeddings preserve essential structural information that is often lost in text extraction. This research underscores that the choice of embedding method should align with the nature of the document to maintain retrieval accuracy, suggesting that multimodal methods are preferable for visual data-intensive queries.