Apple's Ferret AI model is a step towards Siri seeing and controlling apps (appleinsider.com)

Apple has made strides in enhancing Siri's capabilities with the development of Ferret-UI Lite, an innovative AI model designed for local processing on iPhones. This advancement is significant as it places emphasis on user privacy by minimizing reliance on cloud-based large language models (LLMs), which have faced criticism for data security concerns. Ferret-UI Lite, built with 3 billion parameters, is an end-to-end GUI agent that can interact with mobile, web, and desktop environments, allowing Siri to visually comprehend and engage with app interfaces directly. The technical evolution of Ferret-UI Lite is noteworthy, utilizing techniques such as chain-of-thought reasoning and reinforcement learning to improve performance during inference. One standout feature is its zoom-in mechanism, which enhances the model's ability to analyze specific UI elements by cropping images around predicted locations, mimicking human visual attention. While the performance on certain GUI tasks has room for improvement when compared to larger LLMs, Ferret-UI Lite has demonstrated promising accuracy, outperforming some larger models in benchmarks. This research indicates a positive trajectory for developing compact, effective AI agents that can operate seamlessly on personal devices, ultimately paving the way for smarter and more private interactions in mobile technology.