Why Larger Models Learn More: Capacity, Interference, Rare-Task Retention (arxiv.org)

Recent research highlights a critical factor behind the superior performance of larger machine learning models: their ability to learn complex and infrequent tasks that smaller models struggle with. By investigating the effects of model scaling, the study reveals that while smaller models tend to allocate their resources to high-frequency, low-complexity tasks, larger models have a more flexible resource allocation. This flexibility reduces interference during training, allowing larger models to accumulate learning on rare tasks without overwriting previous knowledge. The significance of this finding lies in its implications for model selection and training strategies within the AI/ML community. As confirmed by experiments with OLMo models ranging from 4 million to 4 billion parameters, only the larger configurations demonstrated improved learning on complex tasks. This research not only elucidates why larger models excel in practical applications but also offers valuable insights for future model development, particularly regarding how to optimize training data mixtures for improved learning outcomes.