I made a kernel 2.2x faster. It made my training loop 3x slower (kyrieblunders.bearblog.dev)

A new paper details the development of a fused decode-attention kernel, which achieved a remarkable 2.2× speedup over its predecessor in microbenchmarks for reinforcement learning (RL) training loops. However, the integration into Hugging Face's generation framework inadvertently caused the overall decode step to become nearly 3x slower. This discrepancy highlights the complexities involved in practical implementations, where microbenchmarks do not always translate to real-world performance improvements due to dependencies on different compile paths and execution strategies. The significance of this work lies in its exploration of reinforcement learning post-training, specifically using a policy gradient method known as Generalized Reward Policy Optimization (GRPO), which distinguishes itself by omitting traditional value networks. Instead, it focuses on comparing outputs from sampled prompts, which simplifies the learning process but introduces challenges like length and difficulty biases. The paper provides a detailed technical overview, including iterations on the training loop design, use of Per-Token Loss, and the careful construction of masks to prevent padding errors. These innovations have potential implications for improving the efficiency and effectiveness of RL-based training strategies for large language models (LLMs).