Show HN: Open benchmark for memory layers vs. pasting full chat history (github.com)

A new open benchmark has been introduced for LeapMemory, designed to evaluate the efficiency of memory layers in comparison to traditional methods that rely on pasting the entire conversation history for context. In this benchmark, the same model answers the same questions under two conditions: one where the complete chat history is included (Condition A) and another where only a single recall result is used (Condition B). The results show that LeapMemory significantly reduces the token count required for context, achieving reductions as high as 91.7% without sacrificing accuracy—both conditions attained full scores against expected answers. This benchmark is significant for the AI/ML community as it demonstrates a more efficient way of handling context in conversational AI models, particularly at scale where growing conversation histories can hinder performance due to token limits. The study also allows for end-to-end verification through its open-source framework, encouraging broader testing and validation of memory systems. Additionally, by revealing the effectiveness of direct recall methods over traditional context-pasting techniques, LeapMemory could pave the way for enhanced architectures that enable more sophisticated and efficient conversational agents.