Kandinsky S2V 1.0 – an open family of speech-to-video models (kandinskyavatar.github.io)

Kandinsky S2V 1.0 has been launched, introducing an innovative open family of speech-to-video models designed for controllable audio-driven human animation. The family includes two variants: Kandinsky S2V 1.0 Lite with 2 billion parameters and Kandinsky S2V 1.0 Pro with 21 billion parameters. These models can generate long-form video at 25 FPS from a reference image, speech waveform, and text prompt, ensuring that lip motion, expressions, poses, and gestures align with the audio input. Notably, the Pro version outperforms established open-source and proprietary models, particularly in lip synchronization and across other benchmarks, thanks to its advanced training methods and architectural choices. The significance of Kandinsky S2V 1.0 lies in its capacity to advance the field of speech-to-video generation, combining cutting-edge features such as a shared Qwen2.5-Omni architecture that enhances audio and text conditioning and a novel reference-aware memory management system. By optimizing the training data with 1.17 million selected single-speaker clips and implementing a unique area-adaptive loss function, the models address longstanding challenges in maintaining synchronization and quality. Furthermore, the release of these models and their code under the MIT license fosters open research and innovation in the burgeoning AI/ML community, potentially transforming how animated content is created from audio inputs.