NitroGen: Unified vision-to-action model designed to play video games (huggingface.co)

NVIDIA has announced the release of NitroGen, a unified vision-to-action model that interprets video game footage and translates it into gamepad actions without using traditional reward-based training. Instead, NitroGen employs large-scale imitation learning, harnessing over 10,000 hours of video from human gameplay. Designed primarily for action, platformer, and racing games, NitroGen is less adept at handling genres reliant on mouse and keyboard controls. The project aims to explore whether extensive training on diverse human gameplay can lead to general-purpose embodied abilities, paralleling the emergent behaviors observed in large language models. The implications of NitroGen are substantial for the AI/ML community, potentially revolutionizing game AI, automating quality assurance processes in game development, and contributing to advancements in embodied AI research. Built upon a vision transformer architecture, NitroGen utilizes NVIDIA hardware to optimize performance, achieving quicker training and inference times. With an architecture that combines vision transformers with a diffusion matching transformer, the model processes RGB frames and outputs actions in a structured format, showcasing a significant step forward in bridging computer vision and interactive AI in gaming contexts.