LLM memory systems benchmark: high recall near-zero precision for tested systems (arxiv.org)

A recent study has introduced PrecisionMemBench, a groundbreaking benchmark specifically designed to evaluate the retrieval precision of large language model (LLM) memory systems, independent of the generative models they support. This new benchmark reveals significant shortcomings in existing memory systems, which often report high recall rates while failing to retrieve relevant information effectively, achieving mean retrieval precision rates as low as 0.05 to 0.08. The research emphasizes the necessity of measuring retrieval quality in isolation, highlighting structural flaws in current evaluation methods like LoCoMo, particularly in multi-turn dialogues where topic drift exacerbates retrieval inaccuracies. Moreover, the study introduces Tenure, a local-first structured belief store that achieves perfect retrieval precision (1.0) with impressive latency times of under 15 milliseconds. In stark contrast, other comparison systems struggle, showcasing lengthy ingestion times and zero active retrieval. These findings indicate that relying solely on answer-quality evaluations can obscure critical shortcomings in memory retrieval systems, signaling a need for the AI/ML community to adopt more precise metrics for assessing model performance, especially as they become integral in applications requiring reliable and timely information retrieval.