Inferring User Actions from Screen Recordings to Recommend Better Workflows (arxiv.org)

Researchers introduced InvisibleMentor, a system that analyzes screen recordings of users working in feature-rich apps (like Excel) to detect inefficient routines and recommend better workflows. Unlike assistants that wait for a user prompt or depend on logs/APIs, InvisibleMentor passively reconstructs user actions and the surrounding UI using a vision-language model, then passes that structured representation to a language model which generates high-fidelity, actionable suggestions (e.g., replacing repetitive edits with formulaic or built-in features). In user studies the system accurately identified inefficiencies and produced recommendations participants rated as more tailored, actionable, and helpful for learning than a prompt-based spreadsheet assistant. Technically, the pipeline separates visual reconstruction from suggestion generation: the vision-language stage infers UI state and discrete actions from pixels, enabling use with closed-source or uninstrumented software, while the LLM stage translates that inferred trace into structured guidance and alternatives. This approach advances UI-aware multimodal AI and points to proactive, observation-driven productivity assistants that don’t require explicit user queries. It also raises practical considerations—privacy, consent, and on-device processing—to ensure sensitive screen content is handled safely if deployed.