Transformers Are Multi-State RNNs (huggingface.co)

Recent research has revealed that decoder-only transformers, a dominant architecture in natural language processing, can be understood as finite multi-state recurrent neural networks (RNNs). This conceptual shift redefines how we perceive transformers, suggesting they can function like RNNs but with expandable hidden state sizes. The study introduces a new cache compression technique named TOVA, which efficiently reduces the computational costs of transformer models without sacrificing performance. TOVA not only simplifies existing cache policies but also demonstrates impressive results on long-range tasks, performing nearly as well as full models while utilizing just 1/8 of the original cache size. This breakthrough is significant for the AI/ML community as it addresses one of the major bottlenecks in deploying large-scale transformer models: their memory consumption. By framing cache compression as a means to convert transformers into finite multi-state RNNs, TOVA opens up avenues for more efficient model training and inference, enhancing the practicality and scalability of implementing transformers in real-world applications. The code for TOVA has been publicly released, inviting further exploration and adoption of this novel approach in the ongoing evolution of NLP technologies.