Ferret-UI Lite: Lessons from Building Small On-Device GUI Agents (machinelearning.apple.com)

Researchers have introduced Ferret-UI Lite, an innovative end-to-end GUI agent designed for effective interaction with graphical user interfaces on various platforms, including mobile, web, and desktop. This compact agent, built with a 3 billion parameter model, utilizes a combination of real and synthetic data to enhance its performance. By implementing chain-of-thought reasoning, visual tool usage, and reinforcement learning with tailored reward systems, Ferret-UI Lite demonstrates competitive capabilities among small-scale GUI agents. The significance of this development lies in its potential to advance the field of autonomous agents for UI interaction, particularly for devices with limited computational resources. Ferret-UI Lite has achieved impressive accuracy scores in GUI grounding, with benchmarks such as 91.6% on ScreenSpot-V2, and has shown reasonable success in GUI navigation tasks, attaining rates of 28.0% on AndroidWorld. The insights and techniques documented in this work offer valuable lessons for future research into compact, on-device AI solutions, paving the way for more efficient and versatile GUI agents capable of operating in diverse environments.