Computer-Using World Model (arxiv.org)

Researchers have introduced the Computer-Using World Model (CUWM), a novel approach designed to enhance decision-making in complex software environments, particularly within desktop applications like Microsoft Office. CUWM addresses the challenge posed by the impossibility of counterfactual exploration in digital settings, where incorrect user interface (UI) actions can disrupt important workflows. By predicting the next UI state based on the current state and proposed actions, CUWM employs a two-stage factorization process that first generates a textual summary of state changes, followed by a visual representation that synthesizes the subsequent screenshot. This method allows agents to simulate actions before execution, significantly improving the robustness of their performance. The significance of CUWM lies in its ability to facilitate better planning and learning in environments that are fully digital and deterministic, where traditional trial-and-error methods are impractical. Trained on real user interactions with Microsoft Office, CUWM incorporates a reinforcement learning phase to fine-tune its predictions, ensuring alignment with the structural characteristics of computer-using scenarios. Initial evaluations show that the world-model-guided action predictions lead to improved decision quality across various tasks, potentially paving the way for more intelligent, context-aware software agents in the future.