DiffusionBlocks: Training Neural Networks One Block at a Time (pub.sakana.ai)

Researchers have introduced DiffusionBlocks, a groundbreaking method for training neural networks that allows for block-wise training instead of end-to-end backpropagation. By dividing neural networks into smaller, independent blocks, this approach significantly reduces memory requirements during training, allowing only one block to be processed at a time. This innovative framework retains competitive performance with traditional methods while improving memory efficiency, which could democratize access to AI model development for smaller organizations and individual researchers. Significantly, DiffusionBlocks offers a theoretical foundation by leveraging insights from diffusion models, where each block focuses on gradually approaching the target rather than coordinating with the entire network. This aligns with recent advances in scalable AI, particularly in architectures like Transformers. The method was validated across various domains, demonstrating comparable performance to conventional training methods while drastically lowering memory consumption. Future research avenues include exploring why this efficiency also corresponds with performance improvements and adapting existing large-scale pretrained models to the DiffusionBlocks framework, potentially reshaping the landscape of AI training accessibility.