Model-Based GUI Automation (Springer SoSyM) (link.springer.com)

A new paper redefines visual GUI automation by shifting from traditional process-based scripts to a model-based framework. The author formalizes GUI automation with mathematical models and practical implementations (including the Brobot-originated ideas and a DoT app) to make the environment, actions, pathfinding, traversal and state management explicit. Drawing on software testing, machine vision, robotics and human cognition, the approach addresses the inherent stochasticity of real GUIs, supports integration and unit testing of automation code itself, and demonstrates concrete reductions in code complexity and improved robustness in larger automation tasks. This is significant because it resolves a core mismatch: existing visual automation treats GUIs as deterministic, discrete processes (good for test scripts) while real-world interactions are continuous, adaptive and history-dependent. Unlike pattern-matching, deterministic tools (e.g., OpenCV matchTemplate, SikuliX) the model-based method represents the GUI environment explicitly, enables dynamic decision-making (adaptive goal pursuit), and systematically tests the automation logic—mitigating brittleness and scaling limits (state explosion) typical of event-interaction graph approaches. If adopted, the framework could expand reliable GUI automation into domains previously impractical for visual automation and make automation code itself testable and maintainable.