Airbnb: Agent-in-the-Loop: Data Flywheel for LLM-Based Customer Support (arxiv.org)

Airbnb introduces Agent-in-the-Loop (AITL), a production-ready “data flywheel” that continuously improves LLM-based customer support by embedding four types of human annotations directly into live workflows: pairwise response preferences (which reply is better), agent adoption and rationales (which suggestions agents used and why), knowledge relevance checks (whether retrieved documents are useful), and identification of missing knowledge. Rather than relying on slow, batch offline labeling, these signals are captured during normal support operations and fed back into model updates for retrieval, ranking, and generation—cutting retraining cycles from months to weeks and keeping the system aligned with real-world needs. In a US agent pilot, AITL delivered measurable gains: +11.7% recall@75 and +14.8% precision@8 for retrieval, +8.4% in generation helpfulness, and a +4.5% increase in agent adoption of suggestions. The approach demonstrates a practical pattern for continual learning in deployed LLM systems: seamlessly instrument human workflows to produce high-signal annotations, use them to improve both knowledge retrieval and response generation, and accelerate iteration. For the AI/ML community, AITL is a strong example of operationalizing human-in-the-loop feedback at scale—improving model quality, knowledge curation, and human-AI collaboration while reducing labeling bottlenecks and deployment friction.