Robot-use agents (web.mit.edu)

In 2023, significant advancements are being made in testing Large Language Models (LLMs) as robot-use agents, effectively allowing them to operate robots similarly to how they manipulate digital tools. Traditionally, LLMs were viewed as inadequate for real-time robotic control due to high latency and insufficient spatial and causal understanding. However, emerging models like Fable and Astra are starting to challenge this perception, demonstrating improved capabilities in general-purpose robotic control. Recent demonstrations indicate that LLMs may soon exhibit a level of proficiency that could accelerate the integration of AI within the robotics domain. The implications of this shift could be monumental. With cloud-based intelligence as the driving force, robots could become AI-enabled with mere software updates, transforming any internet-connected device into a potential robotic tool. This contrasts sharply with traditional methods that require extensive customization for each robot type. While there are still challenges, including inherent latency issues and reliability concerns in critical applications, the trend suggests that LLM-controlled robots could foster faster and broader distribution of robotic intelligence. As the possibility unfolds for every digital or physical device to be accessed by agentic AIs, both opportunities and risks for innovation within the AI/ML community are set to multiply.