NeoMME: Multimodal encoders trained from scratch with a single Transformer (huggingface.co)

The recently announced NeoMME introduces a new family of multilingual multimodal encoders, available in 260M and 800M parameter sizes. Unlike conventional models, NeoMME utilizes a single bidirectional Transformer to handle both text tokens and raw image patches without relying on separate pretrained components. Trained from scratch with a masked discrete-diffusion objective, it efficiently encodes images and text concurrently, achieving impressive throughput with 51 pages encoded per second on a high-resolution input. Its design also incorporates hierarchical token pooling and asymmetric quantization, dramatically reducing storage requirements for embeddings while maintaining over 95% of baseline retrieval quality. NeoMME's significance lies in its potential to streamline visual document retrieval by eliminating the need for extensive preprocessing steps like OCR, which can distort the original content's layout and design. The model outperforms existing multimodal retrievers, achieving the highest nDCG@10 score among models under 800M parameters and offering a lower compute overhead. By integrating a unified modal processing approach and leveraging advanced techniques like grouped-query attention and long bidirectional context, NeoMME represents a substantial advancement in building more efficient and robust AI systems for document retrieval, potentially reshaping how we interact with multimodal data. The model is now accessible through Hugging Face Transformers, with all checkpoints released under the Apache 2.0 license.