You could have invented transformers from memory (alexwang.ai)

Recent discussions have highlighted how large language models, like Astra, can adeptly recall and relay information, demonstrating an impressive memory capacity. This exploration focuses on understanding the mechanics behind memory storage in AI models, particularly through linear maps, multi-layer perceptrons (MLPs), and self-attention mechanisms. The ability to memorize and effectively recall information is crucial, as it directly affects model performance. The pivotal metric introduced is the interference-to-signal ratio, which indicates how additional memories can disrupt retrieval accuracy. This study outlines the evolution from simple linear memory stores, which hold limited capacity due to interference, to more advanced architectures like MLPs and self-attention layers that significantly improve recall and capacity. The research emphasizes that while basic linear memory stores face limitations as they approach capacity—about the dimension of the keys—optimizations like the optimal linear memory store can effectively double this capacity by minimizing interference. Furthermore, transitioning to nonlinear memory stores, embodied by MLPs, allows for selective suppression of irrelevant memories during recall, enhancing the overall reliability of information retrieval in AI systems. This work contributes significantly to the AI/ML community by providing insights into the underlying structures that define memory recall in models, potentially guiding future advancements in memory-efficient algorithms and architectures.