Google research: Titans and MIRAS: Helping AI have long-term memory (research.google)

Google Research has unveiled the Titans architecture and the MIRAS framework, two significant advancements that empower AI models with long-term memory capabilities while allowing them to process vast contexts in real-time. Traditional Transformer models, while effective, struggle with scalability due to heightened computational costs as sequence lengths increase. Titans addresses this limitation by integrating a novel, dynamic memory mechanism that actively learns and updates its memory as data flows in, rather than relying on fixed-size memory structures. This is further enhanced by MIRAS, which provides a theoretical foundation for combining new information with previously learned concepts, focusing on selective retention using a "surprise metric" to prioritize significant input. The significance of Titans and MIRAS for the AI/ML community lies in their ability to improve models' performance in handling long-context tasks, such as document understanding and genomic analysis, surpassing existing architectures like Transformers and linear RNNs. By utilizing deeper memory modules and innovative forgetting mechanisms, the Titans architecture demonstrates superior accuracy and efficiency in various benchmarks, including language modeling and commonsense reasoning. The research indicates that these architectures can handle context windows larger than 2 million tokens, marking a pivotal shift toward more sophisticated and responsive AI systems capable of real-time adaptation and understanding of complex, long-form data.