Show HN: WorldDiT – a 399M world-action model for simulated robot tasks (huggingface.co)

WorldDiT has been introduced as a groundbreaking unified diffusion model focused on world and action modeling for robotic tasks, featuring 399 million parameters. This model incorporates continuous action generation alongside an innovative auxiliary future normalized RGB patch prediction within a single diffusion transformer architecture. The initial evaluation of WorldDiT was conducted using the LIBERO benchmark, demonstrating an impressive mean success rate of 94.9% across various robotic tasks, showcasing its potential for efficient language-conditioned robotic manipulation. The significance of WorldDiT lies in its ability to seamlessly integrate multimodal data—combining visual input from multiple camera perspectives and language instructions—to enhance robotic decision-making. Technical aspects of WorldDiT include its use of a 7-step action sequence, a temporal ensembling mechanism, and specialized encoders such as the OpenAI CLIP and MAE ViT-B, all aimed at optimizing performance in simulated environments. This release not only provides four model checkpoints for different LIBERO tasks but also a self-contained inference runtime, encouraging further research and development in the realm of AI-driven robotic manipulation.