The Bitter Lesson of Agent Harnesses (browser-use.com)

A recent blog post discusses a significant shift in the development of agent frameworks for large language models (LLMs), arguing against the abstraction of tools that limit their capabilities. Previously, developers wrapped LLMs in helper functions for tasks like clicking or typing, but this limited their action space. The new approach emphasizes providing LLMs direct access to low-level browser commands, specifically the Chrome DevTools Protocol (CDP), allowing them to handle complex interactions with web pages autonomously. This shift not only simplifies the architecture—reducing thousands of lines of code to under 600—but also empowers LLMs to troubleshoot issues on their own, such as reattaching to fresh targets in case of errors. This change is significant for the AI/ML community as it reveals that LLMs can learn to manipulate complex systems in real-time without rigid constraints imposed by developers. The implications are profound: agents trained with access to CDP can autonomously adapt to problems they encounter, such as file uploads or navigating web interfaces, thereby increasing their utility in real-world applications. As demonstrated, the agent can self-correct and enhance its functionality by writing necessary code on the fly, showcasing a more versatile and efficient use of LLMs in automation tasks.