Show HN: We matched full-context recall on ~1% of the tokens (open benchmark) (github.com)

A new open-source tool called Compresh has been introduced, challenging traditional methods for handling long conversations in large language models (LLMs). Most LLM applications resend entire conversation histories, leading to inefficiencies and diminishing quality due to "context rot" as transcripts grow. Compresh, however, employs a novel strategy: it reconstructs a query-aware slice of prior context needed for each turn, significantly reducing the number of input tokens without sacrificing answer quality. In tests using the EpBench benchmark, Compresh achieved a 66% reduction in tokens sent—dropping from 40.9 million to 13.9 million—while maintaining comparable recall levels to the full-context method. This innovation is significant for the AI/ML community as it addresses one of the critical limitations in processing lengthy dialogues: the degradation of contextual relevance with increased input size. Compresh's approach highlights a shift towards more efficient memory utilization in conversational AI systems, with implications for enhancing user experience and reducing computational costs. The tool's performance improves with longer conversations, suggesting that as discussions become more extensive, the benefits of Compresh over traditional methods become increasingly pronounced. Importantly, this research opens the door for further inquiry into optimizing memory and context management in various applications of LLMs.