NanoGPT Slowrun: 10x Data Efficiency with Infinite Compute (qlabs.sh)

In a groundbreaking announcement, researchers have achieved 10x data efficiency with their new approach, NanoGPT Slowrun, challenging conventional scaling laws in AI training. Unlike the recommendation from Chinchilla for a ~5M parameter model using 100M tokens, their method leverages ensembles—training multiple models independently and aggregating their outputs—which enhances generalization without relying on significantly more data. Notably, they discovered that extending training epochs for individual models worsened their performance, yet ensemble performance improved, highlighting how diverse model training dynamics can lead to better collective outcomes. Key innovations in NanoGPT Slowrun include chain knowledge distillation, which utilizes only the immediately preceding model as a teacher to maintain memory efficiency, and the implementation of loopy transformer architecture that refines predictions by allowing layers to iterate multiple times. These developments not only push the boundaries of what is possible with current architectures but also emphasize the importance of systematic neural architecture search for future efficiency gains. The team's findings not only open the door to significant advancements in model performance but also suggest that further breakthroughs could extend data efficiency potential even further, aiming for a target of 100x within the next year.