Human-Like Neural Nets by Catapulting (gwern.net)

A new speculative proposal suggests creating artificial neural networks (NNs) with human-like performance through a training method called "catapulting." This approach involves using high-learning-rate strategies on overparameterized models, which could unlock true generalization in AI by mimicking the way human brains function. The proposal argues that while current NNs often excel in narrow tasks, they exhibit limitations that biological intelligence does not, highlighting a disparity in how each learns and generalizes knowledge. The concept emphasizes that overparameterization, along with curated small datasets, could facilitate more efficient learning and adaptation in AI models. The significance of this proposal lies in its potential to resolve longstanding discrepancies between artificial and biological intelligence. If successful, catapulted NNs could exhibit superior resilience against adversarial attacks and cloning, improve computational and economic efficiency, and provide more reliable options for AI safety. Testing these ideas may involve training multi-trillion-parameter models with innovative cyclical learning rates, ultimately aiming to bridge the gap between AI and human cognition. This could shed light on the perplexities of intelligence in both domains while paving the way for safer and more general AI systems.