Ferret-UI Lite – Apple on-device end-to-end GUI agent (arxiv.org)

A recent publication has introduced Ferret-UI Lite, a compact, end-to-end graphical user interface (GUI) agent designed specifically for on-device operation across mobile, web, and desktop platforms. This innovative model, boasting a compact size of just 3 billion parameters, leverages a diverse data mixture sourced from both real and synthetic GUI environments. By employing techniques such as chain-of-thought reasoning and reinforcement learning with tailored rewards, Ferret-UI Lite achieves competitive performance levels in the burgeoning field of autonomous GUI interactions. The significance of Ferret-UI Lite lies in its ability to demonstrate that effective autonomous agents can be developed for smaller models, thereby making them more accessible for on-device applications. It achieved impressive performance on key benchmarks, scoring 91.6%, 53.3%, and 61.2% on the ScreenSpot-V2, ScreenSpot-Pro, and OSWorld-G datasets, respectively, and marked success rates of 28.0% on AndroidWorld and 19.8% on OSWorld for GUI navigation. This advancement not only underscores potential improvements in user experience across platforms but also opens new avenues for research into efficient, lightweight AI solutions capable of real-time decision-making in diverse environments.