We taught AI a new modality – UI (computerhumaninteraction.com)

A new AI-driven approach called CHI introduces a novel modality for Large Language Models (LLMs) by teaching them to understand and generate user interfaces (UIs) interactively. Using a custom UI Definition Language (UIDL), CHI enables an LLM to describe the structure of a web UI component tree and react dynamically to user input events. This iterative interaction loop continues until the AI gathers sufficient information to produce outputs that conform precisely to the required target schema, effectively bridging the gap between freeform human communication and rigid data collection needs. Built on top of the Vaadin platform, CHI leverages high-level UI components to maintain interface coherence and accessibility throughout the interaction, ensuring a smooth and user-friendly experience. This innovation is significant for the AI/ML community as it marries natural language understanding with real-time UI generation, allowing AI to guide users through structured data input via conversational yet controlled interfaces. The approach addresses a common challenge: people tend to provide overly detailed or unstructured input, whereas AI systems require clean, schema-compliant data. By dynamically shaping the UI, CHI enables more precise, efficient, and human-centered data acquisition, opening new possibilities for AI-enhanced forms, interactive applications, and adaptive interfaces.