Open source - phone agent automation (github.com)

Open-AutoGLM has introduced a cutting-edge phone agent framework designed to automate tasks on mobile devices using natural language commands. Built on the AutoGLM architecture, the Phone Agent utilizes a combination of visual language models and intelligent planning capabilities to interact with smartphone screens through ADB (Android Debug Bridge). Users can issue requests like “open the app and search for a restaurant,” and the Phone Agent will interpret the command, navigate the interface, and execute the required actions seamlessly. Furthermore, it includes features for sensitive action verification and manual intervention during complex operations like login or CAPTCHA scenarios. This announcement is significant for the AI/ML community as it exemplifies advancements in the integration of NLP (Natural Language Processing) and computer vision in mobile environments, enabling much more intuitive interactions with technology. The framework is optimized for popular Chinese applications and comes in both a standard and a multilingual model, encouraging wider accessibility. It supports over 50 popular applications, allowing developers to create advanced automated solutions for everyday user tasks. The potential implications of such technology include a transformative impact on user experience, enabling devices to understand and act on human intentions more fluidly while promoting greater efficiency in mobile interactions.