Backprop Alternative: Augmented Lagrangian Predictive Coding (pub.sakana.ai)

Researchers have introduced a novel training method called Augmented Lagrangian Predictive Coding (PC-ALM), which serves as a local alternative to backpropagation in deep learning. This method enables the training of residual Multi-Layer Perceptrons (MLPs) with depths of up to 1000 layers while nearly matching the performance of backpropagation. PC-ALM operates by employing layer-local dynamics, where each layer uses feedback control systems to distribute supervision signals across the network, overcoming the signal decay issues traditionally associated with predictive coding, especially in deep, narrow networks. The significance of PC-ALM lies in its potential to enhance energy-efficient deep learning on neuromorphic hardware, as it mimics mechanisms observed in biological brains, which cannot strictly utilize backpropagation. Key technical contributions include the integration of dual neurons that act as Lagrange multipliers, allowing PC-ALM to provide exact credit signals akin to those generated by backpropagation, without requiring synchronized updates. This approach not only facilitates effective training across complex datasets but also introduces a control-theoretic perspective on credit assignment, which may inspire new local learning algorithms in the AI community.