Long-Term Memory for AI:50M-Token Window,Is Faster,Cheaper Than Recompute (arxiv.org)

A recent breakthrough in AI long-term memory capabilities has been achieved with the introduction of a memory layer that supports a 50-million-token context window. Utilizing the galahad-kv public package, this innovation allows large language models to save and retrieve their internal key-value (KV) states from encrypted local NVMe disks without the need for recomputation. When tested on the Gemma 4 models, the system demonstrated remarkable efficiency, achieving speeds 2.8 to 4.3 times faster than recomputing the KV states, while also consuming 8.8 to 12.3 times less GPU energy over extensive token streams. The implications of this development are significant for the AI/ML community as it addresses the limitations of conventional context windows, allowing models to recall information from considerably earlier in the dialogue with high accuracy—82% success for the 12B model and 98% for the 31B model on the retrieval of facts stored millions of tokens prior. This approach not only enhances the models' capacity to handle larger data but also streamlines resource usage, setting a foundation for future advancements in long-context processing and memory-efficient architecture in AI systems.