AI-native Android OS running agents to operate your phone (github.com)

A new AI-native application, currently built for LineageOS 22.2 / Android 15, introduces an innovative "AI phone agent" that enables intuitive control of smartphones through a vision-capable language model (LLM). This assistant operates by capturing the device's screen and analyzing changes using a compatible LLM API, allowing it to execute tasks like tapping, swiping, or entering text—all based on user-defined objectives. The service is task-driven and remains inactive by default, ensuring privacy by not capturing any data until explicitly enabled in the settings. This development is significant for the AI/ML community as it showcases the potential of integrating LLMs with mobile systems, paving the way for more sophisticated AI-driven interactions on personal devices. Key technical details include the application's unique architecture, which leverages a combination of screen capture APIs and input event injection to mimic user actions while providing a responsive, adaptable interface. The project raises interesting implications regarding user privacy, as it necessitates careful handling of screen content and user data, advocating for secure implementations and self-hosted LLM endpoints to mitigate risks.