What do MTEB scores mean, and how should you choose an embedding model? (thottingal.in)

The Massive Text Embedding Benchmark (MTEB) has emerged as a critical tool for evaluating text embedding models, which convert text into numerical representations for various applications, including search systems. The MTEB collects scores across different languages and tasks; however, a recent analysis highlighted significant irregularities in model scores, particularly showing that the leading model, microsoft/harrier-oss-v1-27b, scored 32 points higher in Malayalam than in English. This discrepancy raises questions about the fairness and validity of comparing scores across different languages, as the evaluation metrics are not uniformly applied, with English being assessed on a more complex and varied set of tasks. This investigation reveals that many scores are dominated by specific tasks that do not allow for a direct comparison of model performance across languages. For instance, English's low performance can largely be attributed to a specific challenging task involving low-resource languages, while Malayalam scores well due to a predominance of simpler translation tasks. This highlights potential biases in model evaluation and suggests that developers and researchers should approach the MTEB scores with caution, considering the underlying task distributions and the implications on actual embedding quality for different languages, especially those with fewer resources.