DeepSeek Engram: Conditional Memory via Scalable Lookup (github.com)

DeepSeek has unveiled the Engram module, a novel approach to conditional memory that enhances the sparsity of large language models (LLMs) by enabling efficient knowledge lookup. This development is significant as it introduces an alternative to Mixture-of-Experts (MoE) architectures, addressing a limitation in Transformers that lack a native mechanism for dynamic knowledge retrieval. The Engram module modernizes $N$-gram embeddings for $\mathcal{O}(1)$ lookup, facilitating substantial increases in computational efficiency while improving model performance across various domains, including knowledge retrieval, reasoning, code comprehension, and mathematical tasks. Key technical advancements include the establishment of a U-shaped scaling law that aids in optimal capacity allocation between neural computation and static memory, yielding consistent performance improvements over MoE baselines under strict evaluation conditions. Moreover, Engram's deterministic addressing capability allows for the offloading of extensive embedding tables to host memory without significant inference delays, preserving model efficiency. The light implementation provided showcases the essential functionality of the Engram module, making it an exciting advancement for AI and machine learning researchers looking to enhance LLM capabilities.