Conditional Memory via Scalable Lookup (arxiv.org)

A recent paper introduces "conditional memory" through a novel module named Engram, designed to enhance the capabilities of large language models (LLMs) by optimizing knowledge retrieval. Unlike traditional Mixture-of-Experts (MoE) models that rely solely on computational resources, Engram incorporates a scalable lookup system based on classic $N$-gram embeddings, providing O(1) memory access. By addressing the Sparsity Allocation problem, this approach reveals a U-shaped scaling law that balances neural computation with static memory usage. The efficiency gains are significant, as the Engram module has been scaled to 27 billion parameters and demonstrated superior performance across various reasoning tasks, including gains of up to 5.0 on the BBH benchmark. This development marks a pivotal advance for the AI/ML community, enhancing both the retrieval of knowledge and the general reasoning capabilities of LLMs. Notably, Engram allows for improved attention capacity for long-context tasks while offloading local dependencies to memory lookups. The module's deterministic addressing also enables runtime prefetching from host memory with minimal overhead, establishing a new standard for infrastructure-aware efficiency in sparse models. This innovation positions conditional memory as a critical primitive for future advancements in neural network architectures.