Matrix Orthogonalization Improves Memory in Recurrent Models (ayushtambde.com)

Recent research has showcased a novel approach to enhancing memory in recurrent neural networks (RNNs) by incorporating matrix orthogonalization, a strategy inspired by the highly successful Muon optimizer. Traditionally, architectures like LSTMs excel in associative recall but struggle with noisy environments, leading to the introduction of the mLSTM, which uses a matrix to store memory. While mLSTMs perform well in standard associative recall benchmarks, they lag in noisy associative recall tasks. By orthogonalizing the memory matrix during read operations—without overwriting it—the researchers found a significant boost in performance, especially for complex tasks with larger vocabularies and longer sequences. This improvement is particularly relevant for applications in long-horizon reinforcement learning, where memory efficiency is crucial and the overhead of transformer models is not feasible. Testing revealed that orthogonalized mLSTMs dramatically outperformed their non-orthogonalized counterparts, especially in more challenging scenarios where noise interferes with data. Although these results are promising, the researchers urge caution, noting that the findings are based on small models and synthetic tasks, indicating that further exploration is needed to assess the approach's effectiveness in real-world situations and larger architectures.