Multimodal foundation model for image and video understanding from Microsoft (huggingface.co)

Microsoft has unveiled Mage-VL, an efficient, codec-native multimodal foundation model designed for image and video understanding. This innovative model addresses the Moravec’s paradox in Visual Language Models (VLMs), where existing models excel in offline reasoning but struggle with real-time perception. Mage-VL employs a unique approach by decoding video streams into anchor and predicted frames, strategically retaining only key visual information. This method reduces visual tokens by over 75% while significantly speeding up inference speed by up to 3.5 times compared to traditional uniform frame sampling. Significantly, Mage-VL's architecture combines a from-scratch Codec-ViT visual encoder with a lightweight Qwen3-4B causal decoder. This dual-component design allows for seamless handling of images, videos, and real-time streaming, eliminating the need for multiple models. Notably, Mage-VL demonstrates strong performance on various benchmarks, outperforming models trained on much larger datasets and proving data efficiency through its training on around 100 million unlabeled images and videos. This advancement positions Mage-VL at the forefront of the AI/ML community, promising improvements in both the accuracy and efficiency of multimodal learning, especially for applications requiring real-time streaming analysis.