Are Emergent Abilities of Large Language Models a Mirage? (2023) (arxiv.org)

A recent study challenges the notion of emergent abilities in large language models (LLMs), suggesting these capabilities might be a result of researcher bias rather than inherent properties of model scaling. Emergent abilities—sudden transitions from absence to presence of skills in larger models—have been deemed significant due to their unpredictability and sharpness. However, this research argues that such abilities may simply be artifacts of the metrics used for evaluation. The authors propose that nonlinear metrics can obscure genuine model performance changes, while linear metrics reveal more predictable trends. The implications of this finding are profound for the AI/ML community. By demonstrating that emergent abilities can "evaporate" based on metric selection, the study encourages a reevaluation of how LLM capabilities are measured. This insight promotes a more nuanced understanding of model scaling and could lead to refinement in research methodologies. The authors validate their claims through mathematical modeling and three empirical tests on established LLM families and tasks, indicating that a shift in metrics could reshape our understanding of LLM performances in both language and vision tasks, highlighting the need for rigorous evaluation standards in AI research.