Transformers Are Inherently Succinct (openreview.net)

A recent paper presented at ICLR 2026 highlights the exceptional succinctness of fixed-precision transformers, revealing that they can describe certain languages exponentially more compactly than both linear temporal logic (LTL) and recurrent neural networks (RNNs), as well as doubly exponentially more succinct than finite automata. This finding underscores the expressive power of transformers, which form the backbone of contemporary large language models. By introducing succinctness as a crucial measure, the authors demonstrate that transformers can represent complex languages with significantly smaller descriptions, challenging prior notions about their expressivity compared to other architectures. The paper also has significant implications for computational complexity in the AI and machine learning community. The authors establish that the verification of transformers—such as checking language recognition—is EXPSPACE-complete, indicating that such tasks are computationally intractable under standard complexity assumptions. This complexity arises due to the need for larger structural representations in succinct formalisms like transformers. By employing a fixed precision approach, the research not only improves our understanding of transformers but also emphasizes the challenges involved in formal analysis and verification of these models in practical applications.