Bringing Your Muse to Life (research.meta.ai)

Today, Meta announced the launch of Muse Realtime Avatar, an innovative technology that creates interactive, expressive avatars driven by real-time voice. By integrating Muse Realtime Voice, which generates speech tokens that capture both content and delivery, Muse Realtime Avatar produces synchronized visual performances—allowing characters to display subtle expressions and gestures as they participate in live conversations. This advancement enables animals and everyday objects to be animated in ways that maintain their uniqueness while effectively engaging users. The significance of Muse Realtime Avatar lies in its robust technical foundation, leveraging a Diffusion Transformer model that generates videos in real time while maintaining visual consistency and coherence. Enhancements such as a high-quality teacher-student model training approach drastically reduce computation time, allowing avatar generation at subsecond latency while supporting multiple concurrent sessions. With optimized video inference and serving infrastructure, the system streamlines performance to deliver lifelike avatar interactions at scale. The integration of safety measures, including an invisible watermark for traceability, underscores Meta's commitment to responsible AI development as it pioneers new frontiers in embodied AI experiences.