Memory Caching: RNNs with Growing Memory (arxiv.org)

A recent study introduces "Memory Caching" (MC), a novel technique that enhances recurrent neural networks (RNNs) by enabling them to grow their memory capacity in line with sequence length. This development addresses a significant limitation of traditional RNNs, which have fixed-size memory and struggle with recall in longer contexts compared to Transformers. By caching memory states, MC achieves a balance between the linear complexity of RNNs and the quadratic complexity of Transformers, proposing four variants that incorporate mechanisms like gated aggregation and sparse selection. The significance of Memory Caching lies in its potential to improve the performance of RNNs on language modeling and long-context understanding tasks. Experimental results reveal that while Transformers still lead in accuracy, the MC variants significantly close the performance gap and outshine many state-of-the-art recurrent models. This advancement could motivate further research into efficient recurrent architectures and their applications, suggesting a new direction for enhancing sequence modeling techniques in the AI/ML community.