Lumine: A 7B VLM That Plays Genshin Impact for 5 Hours Using Keyboard/Mouse (danhussey.bearblog.dev)

Lumine, a newly announced 7 billion parameter model, has successfully demonstrated the capability to play Genshin Impact for five hours using keyboard and mouse controls, marking a significant breakthrough in real-time generalist gaming AI. Unlike previous models, Lumine processes raw game pixels at 5Hz while outputting actions at up to 30Hz, allowing it to engage with intricate game mechanics across multiple genres. Its training involved over 1,700 hours of gameplay footage and additional phases focusing on instruction tuning and reasoning, which equips it with the ability to transfer learned skills, such as UI interaction and combat, to other games like Honkai: Star Rail and Wuthering Waves. Despite its impressive performance and human-expert completion times for missions, Lumine faces notable limitations, particularly in reasoning quality and memory, leading to errors in task execution. The model struggles with maintaining long-term focus, resulting in missteps during prolonged missions and perceptual challenges like misinterpreting objects or environments. Suggestions for future improvements include multi-game training to enhance generalization, hierarchical memory structures for better goal management, and leveraging stronger base models to reduce perception errors. This innovative approach underscores the evolving landscape of AI in gaming and raises questions about the potential for more sophisticated generalist agents in real-world applications.