CODA: Rewriting Transformer Blocks as GEMM-Epilogue Programs (arxiv.org)

Researchers have introduced CODA, a novel GPU kernel abstraction designed to enhance the efficiency of Transformer models by reinterpreting their computations as GEMM-plus-epilogue programs. This approach addresses a critical bottleneck in deep learning, where a significant portion of training time is consumed by memory-bound operations rather than arithmetic computations. By allowing these computations to occur while output tiles remain on chip, CODA reduces the need for extensive data movement, which is often a performance drag in existing frameworks. The significance of CODA lies in its dual promise of retaining the efficiency of expert-written GEMM routines while offering a composable set of primitives for various operations within Transformer blocks. This enables a seamless integration of high-level framework productivity with low-level hardware performance. Initial results indicate that both human-crafted and LLM-generated CODA kernels achieve impressive performance across standard Transformer workloads, suggesting that this method could represent a transformative advance in optimizing training stacks for AI and ML applications.