M* (M-Star): A Modular, Extensible, Serving System for Multimodal Models (mstar.stanford.edu)

The recent announcement of M* (M-Star), a modular and extensible serving system for multimodal models, marks a significant advancement in the AI/ML community. Unlike traditional LLM serving systems, which rely on a single autoregressive loop for inference, M* accommodates the diverse architectures of contemporary composite models like BAGEL and Qwen3-Omni. These new multimodal models consist of various components—such as vision encoders and audio codecs—operating in adaptable loops and parallel paths based on the input data. M* allows for a flexible composition of these components through its innovative Walk Graph structure, optimizing performance across various modalities and tasks. M* excels where current systems fall short, effectively addressing the challenges of architectural diversity and modularity. Benchmarks indicate that M* outperforms specialized systems by significant margins—up to 2.7× for speech and image processing and 12.5× for world-model rollouts. The runtime autonomously manages crucial elements like placement and batching, enabling seamless integration of complex behaviors like diffusion loops or classifier-free guidance without requiring bespoke coding for each model. This not only enhances efficiency but also paves the way for developers to create more advanced and varied applications in the multimodal AI landscape.