Agent-browser equivalent for desktop and mobile apps (agent-device.dev)

A new device automation CLI called "agent-device" has been launched, designed specifically for automating and verifying interactions with native mobile and TV applications across iOS, Android, and other platforms. Inspired by browser automation, this tool allows AI agents to effectively understand and interact with user interfaces by capturing only the necessary information, utilizing a structured accessibility tree, and assigning stable references for UI elements. This capability ensures agents can perform tasks such as filling in forms or clicking buttons with semantic precision, which enhances both readability and resilience over time. The significance of this development lies in its promise to standardize automation workflows across different platforms while enabling seamless transitions between development simulators and physical devices. Moreover, the tool supports exploratory automation by allowing agents to record interactions, capture logs, and generate deterministic replay scripts, making it a robust solution for end-to-end testing. With features for debugging, visual proof capture, and even environment manipulation, the agent-device not only streamlines the automation process but also provides deeper insights into automated testing scenarios, positioning itself as a crucial resource for the AI/ML community focusing on application verification and quality assurance.