Apple researchers develop on-device AI agent that interacts with apps (9to5mac.com)

Apple researchers have unveiled Ferret-UI Lite, a lightweight on-device AI model optimized for mobile user interface (UI) interactions. Despite having only 3 billion parameters, it matches or surpasses the performance of larger models with up to 72 billion parameters in GUI navigation tasks. This development is significant as it addresses the limitations of existing GUI agents, which are often too large and resource-intensive for mobile applications. Ferret-UI Lite is designed to operate efficiently on devices, ensuring user data remains private, as it processes interactions locally instead of relying on cloud computing. The model employs innovative techniques, including real-time cropping and zooming to enhance its understanding of specific GUI elements, and it generates its own synthetic training data through a multi-agent system that simulates real-world interactions. While Ferret-UI Lite excels in short, straightforward tasks, it faces challenges with complex, multi-step interactions. Nevertheless, its launch points towards a future where mobile AI agents can provide more autonomous, efficient user experiences without sacrificing user privacy or requiring extensive computational resources.