Helix World: A real-time world model that generates video and stereo sound (noiz.ai)

Noiz AI, in collaboration with The Hong Kong University of Science and Technology, has unveiled Helix World, a groundbreaking real-time interactive audio-visual world model that generates video and corresponding stereo sound simultaneously at 24 frames per second using a single NVIDIA H800 GPU. Unlike existing models that are primarily silent or offer delayed audio, Helix World adapts sound dynamically to the user's camera movements and actions, creating a truly immersive experience. This model addresses crucial shortcomings of prior world models by ensuring sound fields align with camera angles and movements, thereby enhancing realism and interactivity. The significance of Helix World lies in its innovative approach to combining audio and visuals, filled with detailed technical achievements. Helix World scored 79.9 on the WBench navigation split, outperforming other real-time models while effectively maintaining speed with a real-time factor (RTF) of 0.77. It also set a new benchmark with its audio-visual consistency score of 41.8 on HelixBench, far surpassing the only comparable model, EchoWM, which scored 12.7. The detailed training process included filtering and captioning 4,100 hours of video, emphasizing diegetic sound, which plays a crucial role in creating an immersive experience. With an interactive demo available and promising performance metrics, Helix World represents a significant advancement in the field of audio-visual generation, offering exciting implications for gaming, virtual reality, and beyond.