Hopfield and Bam – Neural Networks You Can Train and Use on Pen and Paper (blog.demofox.org)

John Hopfield and Bart Kosko's classic neural networks—Hopfield and Binary Associative Memory (BAM)—are regaining attention for their simplicity and educational value, allowing training and inference with just pen, paper, and integers. Hopfield networks are designed to retrieve the closest stored pattern from noisy inputs, while BAM networks can associate input-output pairs, mapping potentially corrupted patterns back to their original forms. The fundamental mechanism of these networks involves converting binary patterns into bipolar representations, performing matrix operations, and applying a sign function for output stabilization and convergence. The significance of this resurgence lies in the demonstration of their enduring relevance within the AI/ML community, especially as they connect to modern architectures. Notably, a single update in contemporary continuous Hopfield networks parallels the self-attention mechanism in Transformers, bridging classical and cutting-edge concepts. However, as these networks face limitations regarding the density of patterns they can accurately recall, they serve more as pedagogical tools than practical solutions in larger-scale AI applications, prompting researchers to shift towards newer models for complex tasks.