We gave our agent memory: building an LLM Wiki over sources that never sit still (engineering.taktile.com)

A team has developed a persistent memory system for their AI agent, enabling it to maintain an evolving knowledge layer, dubbed an "LLM Wiki." Initially employing a standard retrieval-augmented generation approach that was costly and inefficient, they shifted to compiling insights into a reusable knowledge structure. This approach allows the agent to synthesize answers from previously learned information instead of deriving them from scratch, significantly reducing response times and costs. However, the challenge lies in managing shared sources that are constantly changing, necessitating a verification process for the information accessed. The LLM Wiki operates as a directory of markdown files that are version-controlled, with a classification system that scopes relevant sources to streamline information retrieval. While early tests show that traditional retrieval methods like BM25 are much faster, the team found that using LLM judgment yielded the highest accuracy, albeit at a substantial latency cost. Moving forward, they aim to optimize this by storing fingerprints of sources to minimize unnecessary validations and enhance the efficiency of their memory system for real-time applications. Ultimately, they seek to prove that this evolving knowledge layer can maintain its integrity and utility as it scales.