Looped Transformers as Programmable Computers (2023) (arxiv.org)

Recent research has explored the concept of using transformer networks as universal computers by programming them with specific weights and placing them in a loop structure. This innovative framework allows input sequences to function like punchcards, containing instructions and memory required for data operations. The study demonstrates that a constant number of encoder layers in transformers can replicate essential computing blocks, such as embedding edit functions, program counters, and conditional branches. Through this approach, a 13-layer transformer is capable of emulating small instruction-set computers and executing various programs, including basic calculators and linear algebra libraries. This development is significant for the AI/ML community as it underscores the versatility and potential of transformer architectures, even at lower depths. The ability to execute generalized programming tasks via transformers highlights a paradigm shift in how we perceive the application of neural networks beyond traditional tasks, expanding their utility in computational contexts. This research not only contributes to the ongoing dialogue about the capabilities of transformer models but also opens pathways for creating more sophisticated algorithms that leverage the inherent structure of these networks to function as programmable computers.