If You Need a New UI for Every AI Use Case, You're Doing It Wrong (fsilavong.github.io)

LLMs have largely removed the label bottleneck, but the real limiter for production agentic systems is now domain knowledge capture. Instead of annotating datasets before training, teams are annotating after the model — in prompts, tools, guardrails and evaluations — yet tooling hasn’t kept pace. Developers compensate with bespoke UIs (Streamlit, Gradio) and ad-hoc channels (email, Zoom, DMs), which work for single pilots but fragment feedback, trap SME knowledge, duplicate effort, and prevent scaling beyond a handful of use cases. The proposed fix is a single conversation surface that serves many agents and funnels structured feedback into an evaluation-to-release loop: user/SME messages → agent responses + feedback schema → centralized telemetry & evaluations → prompt/tool updates → versioned releases visible to pilot cohorts. Key technical primitives include structured feedback (thumbs+reason, rubrics, error tags, suggested responses, tool/run traces), assignment and cohorting, experiment tracking with prompt/tool versioning tied to offline/online metrics, and portability across use cases. This architecture closes the loop so teams can measure which changes moved which metrics for which cohorts, iterate faster, and scale to 10+ agents with the same resources — shifting the bottleneck from model prototyping to systematic knowledge capture and governance.