Computer Use Protocol – AI agents can perceive and interact with any desktop UI (github.com)

The Computer Use Protocol (CUP) has been announced as a universal schema designed to enable AI agents to perceive and interact with desktop user interfaces (UIs) across multiple platforms, including Windows, macOS, Linux, and mobile systems. This protocol introduces a compact text encoding that is approximately 97% smaller than JSON, optimizing it for large language models (LLMs) by fitting complex UIs into context windows more efficiently. CUP standardizes the representation of UI accessibility trees, eliminating the need for separate translation layers for each platform and allowing developers to write agent logic once and deploy it anywhere. CUP's technical architecture features a JSON envelope format based on ARIA-derived roles and 15 standardized action verbs that dictate how an AI agent can manipulate UI elements. SDKs for various programming languages facilitate the capture of native accessibility trees, normalization into the CUP format, and execution of actions. By providing a unified framework for UI interaction, CUP promotes greater interoperability among AI tools, enhancing their ability to perform tasks seamlessly across different operating systems while preserving essential native properties. This innovative approach not only streamlines development but also supports the evolving capabilities of AI agents, positioning CUP as a significant advancement in the AI/ML landscape.