Language Models Suck at Real-Time Control (lucrbvi.com)

🤖 AI Summary
A recent experiment using the Celeste Classic game has revealed that large language models (LLMs) perform poorly in Real-Time Control (RTC) tasks, with their capabilities falling significantly short of human performance. The benchmark, dubbed CelesteBench, was created to measure LLMs' abilities in completing game levels under time constraints. The top-scoring model, GPT-6 Astra, only managed to clear 5.17% of the first room, while average human players score around 10% in the same time frame. Most models struggled to progress even in RTC mode, often taking far longer to perform simple actions, such as stacking bowls, than a human would. This finding is significant for the AI/ML community as it highlights the current limitations of LLMs in executing real-time robotic tasks despite their advancements in other areas like coding. The experiments indicate that higher reasoning times and the inability to translate knowledge into action hinder model performance. The results provoke deeper scrutiny over the architectures and training processes of these AI systems, signaling that further research and development are essential before LLMs can effectively engage in time-sensitive real-time control applications. CelesteBench is now open for others to benchmark their capabilities against these models, offering a unique insight into the ongoing evolution of AI realism and robotics.
Loading comments...
loading comments...