Torchwright: Compile computation graphs into vanilla transformer weights (ood.dev)

A new tool called **Torchwright** has been developed to compile computation graphs directly into transformer weights, eliminating the need for conventional training methods. The approach centers around constructing exact weights for transformers to implement specific algorithms, which has been a question in the AI community concerning the limitations of transformers in expressing arithmetic operations. By creating a compiler that translates ordinary Python code into transformer weights, Torchwright enables users to explicitly define algorithms without relying on the uncertainties of model training. This development is significant because it addresses foundational questions about the capabilities of transformer architectures and their inherent limitations. By demonstrating that transformers can execute predefined algorithms with computed weights, the tool challenges assumptions about the necessity of training for effective model performance. The architecture used is deliberately minimalist, initially leveraging feed-forward networks (FFNs) and ReLU activations before expanding compatibility to more conventional transformer architectures, such as Phi-3. Key innovations include memory management through a shared residual stream and the adoption of position-independent attention mechanisms, which maintain the integrity of operations. Ultimately, Torchwright not only advances the understanding of transformer capabilities but also provides a flexible framework for experimenting with various computational graphs in a more intuitive way.