What 1,192 conversations taught us about knowledge base search in AI agents (www.kapa.ai)

Kapa, a company developing AI assistant solutions for customer-facing applications, analyzed 1,192 user interactions with their newly launched AI agent and uncovered significant insights about knowledge base searches. The agent is designed to assist users with technical queries about their product deployments, utilizing 30 native tools and one primary knowledge base search function. Surprisingly, the knowledge base tool emerged as the most utilized feature, often serving as a fallback when users posed questions beyond the scope of the native tools—making up 32.1% of total inquiries. This revealed that users increasingly view the agent as a comprehensive resource rather than a strictly analytics-focused tool. The findings highlight three key roles of the knowledge base: it serves as a failover for unanswered queries, it provides essential context to native tool outputs, and intriguingly, it aids the agent in making informed decisions about which tools to employ in various scenarios. For instance, in cases where users asked for data analyses that didn’t correspond to existing filters, the agent relied on documentation to bridge the gap and properly utilize its tools. These insights emphasize the importance of an effective knowledge base in enhancing AI agent capabilities, suggesting that robust documentation tools could significantly improve user experience and engagement.