Anthropic publicly releases AI tool that can take over the ' mouse cursor(2024) (arstechnica.com)

Anthropic has launched a new AI tool called "Computer Use," which allows users to remotely control their mouse cursor and carry out basic tasks on their computer. Currently available through the API of the mid-range 3.5 Sonnet model, the tool enables users to give multi-step commands that Claude executes by analyzing screenshots and calculating pixel movements to click or type, thereby facilitating task automation. This advancement is notable for the AI/ML community as it represents a significant capability in UI automation, pushing the boundaries of how AI can interact with existing software. However, the tool does have some limitations, such as reliance on sequential screengrabs instead of live video, which can lead to issues like missing transient notifications. Furthermore, it has been described as "cumbersome and error-prone," with some amusing anecdotes from testing, like a distraction to view photos of Yellowstone. As competing companies like OpenAI work on similar tools, the release of "Computer Use" signals an emerging arms race in AI-driven automation, highlighting the potential for significant revenue generation as these technologies mature.