We taught a small LLM to throw away 68% of our RAG context (www.kapa.ai)

Kapa has developed a method to improve the efficiency of AI assistants that respond to complex queries over extensive knowledge bases by incorporating a pruner LLM (language model) to filter context. This innovative approach allows the pruner to drop approximately 68% of the retrieved chunks of information while still maintaining 96% recall of the essential data needed to generate accurate answers. The pruner functions seamlessly between the retrieval and generation stages, significantly reducing costs tied to unnecessary data processing—effectively cutting the query cost by a third. This development is significant for the AI/ML community as it addresses the challenge of managing large volumes of data without sacrificing the performance of the AI models. Traditional methods of relevance scoring often struggled with contextual dependencies between chunks, leading to inefficient data processing. Kapa's method utilizes a listwise approach, where the pruner assesses all retrieved chunks together in relation to the question, making it capable of discerning and retaining relevant information that might otherwise be overlooked, ultimately enhancing the effectiveness and affordability of deploying AI-driven tools.