Why Vector Search fails at LLM memory (and a benchmark to prove it) (github.com)

A new benchmark called PrecisionMemBench has been introduced to assess the performance of large language model (LLM) memory systems, addressing limitations of traditional single-turn evaluation metrics. This benchmark evaluates four critical properties: retrieval precision, noise isolation, session-turn latency, and belief mutability over a set of 89 test cases. By examining these dimensions, PrecisionMemBench reveals how well memory systems perform across multi-turn interactions and whether new beliefs introduced during an ongoing session affect the retrieval process. The significance of PrecisionMemBench lies in its ability to provide a more comprehensive understanding of LLM memory systems, which have struggled with challenges like noise contamination and degradation under session load. Early results indicate that many existing systems, including popular vector storage solutions, fail to maintain precision and noise isolation in multi-turn scenarios. For example, the benchmark shows that while some systems like 'tenure' achieve perfect precision and recall, others, including multiple vector systems, score excessively low on precision, often returning numerous irrelevant beliefs alongside correct ones. This highlights the need for improvements in LLM architectures to enhance their reliability and usability in real-world applications.