Phi-4-reasoning-vision and the lessons of training a multimodal reasoning model (www.microsoft.com)

The announcement of Phi-4-reasoning-vision-15B, a 15 billion parameter open-weight multimodal reasoning model, marks a significant advancement in the AI/ML community by offering a more efficient and capable alternative for a variety of vision-language tasks. This model not only performs exceptionally well in tasks like image captioning and understanding user interfaces but also excels in math and scientific reasoning. Its streamlined design balances reasoning capabilities with low computational costs, achieving competitive performance with heavy-hitting models while requiring significantly less training data—just 200 billion tokens compared to over 1 trillion used for other models. Key technical contributions from the development of Phi-4-reasoning-vision-15B include lessons on model architecture and data quality. The model employs a mid-fusion architecture, which optimally combines visual and textual information while reducing computational demands. Extensive data curation ensured high-quality input, utilizing a mix of open-source and domain-specific datasets. The results of experimental studies highlight the effectiveness of dynamic resolution vision encoders and careful data proportioning in improving performance across various tasks. This focus on fostering smaller, robust multimodal models sets a promising precedent for future AI developments, aiming for both accessibility and high performance in resource-constrained environments.