FLAT: Flash Linear Attention on TPUs (trainstation-project.github.io)

Google has announced the release of FLAT (Flash Linear Attention on TPUs), a set of innovative Pallas implementations designed for efficient training of linear attention layers on TPU v6e chips. Unlike common GPUs, which have robust support for rapid linear attention computations through CUDA kernels, TPUs lacked equivalent capacities, particularly for training. FLAT circumvents these limitations by optimizing the processing through several novel strategies, such as batching multiple attention heads and performing crucial operations on the TPU's matrix unit (MXU) instead of the slower vector unit (VPU). This leads to training speed improvements, with FLAT achieving times as low as 1,982 microseconds compared to 6,106 microseconds for GPU solutions. This development is significant for the AI and machine learning community as it enhances the ability to leverage TPUs for linear attention models, which are increasingly incorporated into the latest large language models (LLMs) such as Kimi K3 and Qwen 3.5. It highlights a growing trend in the AI field toward optimizing hardware utilization for next-generation architectures. The work not only enriches the ecosystem of tools available for model training but also provides critical insights into overcoming architectural challenges, potentially setting a new standard for TPU optimization techniques and expanding options for AI researchers and developers.