Hermes Agent Background Computer Use on macOS via SkyLight Private SPIs (hermes-agent.nousresearch.com)

The recent release of the Hermes Agent introduces a groundbreaking feature for macOS users by enabling background control of the desktop without altering the user's visible cursor position or keyboard focus. This means that while users interact with their machine, the Hermes Agent can perform tasks such as clicking, typing, and scrolling silently in the background using various AI models, including Claude and GPT. This novel approach leverages macOS's SkyLight private SPIs, specifically designed to send synthetic events directly to applications without interfering with the user's current workflow. This innovation is significant for the AI/ML community as it offers greater integration possibilities for AI-driven automation tools on macOS, reducing reliance on device-specific schemas and enhancing usability. Key technical advancements include the use of an open-source driver called cua-driver, which operates through advanced command protocols and security measures to ensure safety during automated tasks. The system includes multi-layer guardrails to prevent harmful actions and offers impressive token efficiency in processing screenshots, further optimizing performance. This development paves the way for more intuitive human-AI collaboration in personal computing environments.