Show HN: SoMatic – Vision-based OS automation framework for AI agents (github.com)

SoMatic has unveiled a new vision-based CLI framework designed for desktop UI automation, leveraging a local YOLO model to precisely detect and number interactive elements in screenshots. This innovative approach provides AI agents with a structured coordinate map to execute actions without ambiguity, targeting elements either by identification number, proximity, or pixel coordinates. Commands in SoMatic return information in JSON format, which streamlines interactions across various applications including native apps, web browsers, and PDFs. This framework's significance lies in its ability to enhance the capabilities of AI agents in automating tasks on desktop environments. By using YOLO for object detection, SoMatic improves the reliability and accuracy of UI interactions, as demonstrated through evaluations against benchmarks. The results suggest that integrating visual detection significantly boosts performance, especially for advanced AI models. Furthermore, the framework is developed under a flexible licensing model, adopting an MIT core while managing AGPL-licensed components separately, ensuring broad accessibility for developers in the AI/ML community who wish to leverage such automation capabilities.