Show HN: mobile-agent – Let AI agents control Android and iOS devices (github.com)

A new tool called mobile-agent has been announced, allowing AI agents to control Android devices, Android emulators, iOS Simulators, and physical iPhones. This tool operates using a daemon that creates a live mirror of every attached screen, managing taps and observations efficiently by updating information in real-time without the need for extensive reading or delays. On Android, it runs as a single-dex agent over adb without needing to install APKs. For iOS, it utilizes the Simulator's HID service and includes a specialized Rust reader to capture accessibility data, eliminating the need for XCTest runners or additional installations. The significance of mobile-agent lies in its potential to streamline the automation of mobile devices for AI applications. By functioning without extensive modifications to the operating system or app installations, it offers developers a novel way to integrate AI capabilities into mobile tasks. The experimental nature of mobile-agent means it utilizes private and hidden interfaces from both Apple and Android, making it a valuable tool for developers looking to explore advanced automation and integration in AI/ML workflows. Currently, it is open-source under the MIT License and is not yet available for public distribution, requiring developers to build it from source.