Blog FlexAttention: The Flexibility of PyTorch with the Performance of FA (pytorch.org)

PyTorch has introduced FlexAttention, a new API that enhances its attention mechanism by marrying the flexibility of PyTorch with the optimized performance of FlashAttention. Traditional attention implementations often limit researchers to existing kernels, hampering innovations in developing new attention variants. FlexAttention tackles this challenge by allowing users to define custom functions for modifying attention scores in just a few lines of idiomatic PyTorch code, significantly reducing the need for custom kernels while maintaining high efficiency. This development is significant for the AI/ML community as it opens the floodgates for experimentation with diverse attention variants, such as Relative Positional Embeddings and Causal Masks, without the cumbersome overhead of memory allocation and computational inefficiencies. The API leverages PyTorch's autograd for automatic backpropagation and employs a technique to efficiently compute attention scores, taking advantage of sparsity. With FlexAttention, researchers can explore novel attention mechanisms free from the constraints of existing implementations, promising to accelerate advancements in deep learning applications and model architectures. The capability for rapid experimentation and performance efficiency is poised to drive innovation across AI model training and deployment.