Trading off compute for memory with activation checkpointing (blog.janestreet.com)

A new method for optimizing activation checkpointing in PyTorch has been developed by summer intern Arsh Koneru, aiming to enhance the balance between memory usage and computing time during model training. Traditional activation checkpointing typically saves model activations during the forward pass for use in the backward pass, leading to significant memory peaks. Koneru's approach introduces a "planning" phase that evaluates the complete computational graph to determine which nodes can be safely dropped, thereby reducing peak memory requirements while keeping additional computation to a minimum. This algorithm counts on a greedy strategy to iteratively identify the best nodes to discard based on their recompute cost and memory savings. The significance of this advancement lies in its potential to alleviate memory constraints during training of large models, which is a critical issue in AI/ML workflows. By smoothing out memory usage over time and leveraging PyTorch's existing capabilities, such as min-cut partitioning, this new method offers a promising baseline for future optimizations. While there is still room for improvement, including addressing dynamic shapes and refining memory estimates, the initial results indicate superior performance over existing methods, marking a significant step forward in resource-efficient model training.