Transformers Are Inherently Succinct (2025) (arxiv.org)

A new study highlights the inherent succinctness of transformers in representing formal languages, demonstrating that these models can express complex concepts far more efficiently than traditional representations like finite automata and Linear Temporal Logic (LTL). This research introduces succinctness as a key measure of a transformer's expressive power, making a strong case for transformers' versatility and efficiency in handling language and computational theory tasks. This finding is significant for the AI/ML community as it not only enhances the understanding of transformers' capabilities but also suggests that properties of these models are challenging to verify, being provably EXPSPACE-complete. This intractability raises important implications for developers and researchers, indicating that while transformers offer powerful modeling capabilities, they also require careful consideration regarding the complexity of their verification. As transformers continue to play a pivotal role in advancing natural language processing and other AI applications, understanding their succinctness could foster innovations in model design and application efficiency.