Generating Automatic Feedback on UI Mockups with Large Language Models (dl.acm.org)

Researchers built a Figma plugin that uses GPT-4 to perform automated heuristic evaluations of static UI mockups by combining designer-selected guideline texts (e.g., Nielsen’s heuristics) with a compact JSON representation of the UI (hierarchy, semantic labels, text, location, size, color). The LLM is prompted with the guidelines and the UI JSON and returns constructive, rule-based suggestions phrased according to effective feedback principles (expected standard, gap, fix). The plugin highlights implicated elements in the screenshot, links to Figma layers, and lets designers hide unhelpful suggestions—feedback which is fed back into subsequent evaluations. Because of context-window limits the system evaluates one static screen at a time; authors evaluated several models and focused on GPT-4 as the best performer. In user studies (3 experts rating 51 UIs, 12 experts’ manual comparisons on 12 UIs, and a 12-expert iterative refinement study), GPT-4 reliably caught many issues—especially straightforward, rule-checkable problems—and designers found the tool practically useful despite occasional inaccuracies. Limitations include hallucination risks, degradation after multiple refinement iterations, sensitivity to mismatches between the JSON and visual/semantic reality, and inability to handle multi-screen/interactivity. The takeaway: LLMs can augment heuristic evaluation today by accelerating feedback and catching subtle issues, but they aren’t a replacement for human evaluators; improving visual grounding and context handling will expand their utility.