MTEB v2: Evaluation of embedding and retrieval systems for more than just text (huggingface.co)

MTEB v2 is a major refactor and feature release of the Massive Text Embedding Benchmark that modernizes the API, adds multimodal and non-embedding retrieval support, and improves reproducibility and usability. The release consolidates many community contributions, standardizes typing and file names, upgrades documentation, and introduces an easier evaluation path (mteb.evaluate) plus a ResultCache for local/remote result management. These changes make MTEB simpler to adopt, extend, and maintain while reducing the risk of breaking changes as the benchmark grows beyond text-only tasks. Technically, v2 defines clear model protocols: an EncoderProtocol that accepts BatchedInput supporting text, images (and soon audio), a CrossEncoderProtocol for cross-encoders, and a SearchProtocol for indexing/searching (with automatic wrappers for encoders and cross-encoders). Retrieval and reranking tasks are unified under a new task schema that includes corpus, queries, qrels and top_ranked lists for reranking evaluation. MTEB now exports descriptive statistics for every task, supports saving model predictions for error analysis, and fully supports Datasets v4 (Parquet reuploads). For users migrating from v1, mteb.evaluate + ResultCache replace the older MTEB.run and load_results workflows, enabling cached, incremental, and multimodal evaluations without re-running published results. Overall, v2 broadens MTEB from an embedding benchmark to a versatile evaluation platform for modern multimodal retrieval and reranking systems.