Show HN: Tine – Drive Wayland Around with Agents (github.com)

Tine, a new command-line tool, has been introduced to enable AI agents to control GNOME Wayland desktops directly. This innovative solution bypasses traditional user consent dialogs and integration issues often experienced with Wayland, making it particularly significant for the AI/ML community, especially for developers working in environments that have been challenging for automation. Tine operates by reading the screen and accessibility trees via AT-SPI2 and injecting keyboard and mouse events at the kernel level, which allows for seamless interaction without the constraints of Wayland's GUI prompts. Key functionalities include the ability for agents to describe the screen state, interact with UI elements based on their accessibility references, and even utilize OCR to identify text regions for precise automation tasks. This tool sets itself apart in the Linux ecosystem by combining structured UI reading with portal-free input methods, representing a major step forward for automation in Wayland environments, which have been historically more locked down compared to X11. Currently in alpha, Tine is specifically designed for Wayland under GNOME and targets a niche that other existing tools have yet to address effectively.