Residual Matrix Transformers: Scaling the Size of the Residual Stream (arxiv.org)

Researchers have introduced the Residual Matrix Transformer (RMT), a novel architecture that redefines the way information is managed within the residual stream of traditional transformers. By replacing the conventional residual mechanism with an outer product memory matrix, the RMT can independently scale the size of this stream, leading to significant improvements in performance without a proportional increase in computational resources. Specifically, the RMT achieves the same loss as standard transformers while utilizing 58% fewer FLOPS, 25% fewer parameters, and 41% fewer training tokens. This advancement is particularly significant for the AI/ML community as it not only enhances efficiency but also improves the variance propagation properties of the model, which is critical for training stability and performance. The RMT outperforms typical transformer architectures in downstream tasks, opening new avenues for more efficient model development. This research underscores the potential for innovative designs in transformer architectures to yield substantial gains, suggesting a shift in how researchers might approach memory and computation in AI models moving forward.