Regressions on benchmark scores suggest frontier LLMs ~3-5T params (aimlbling-about.ninerealmlabs.com)

In a recent discussion on the Latent Space podcast, the Artificial Analysis team highlighted a compelling correlation between the parameter counts of large language models (LLMs) and their performance on the AA-Omniscience Accuracy benchmark. Their analysis indicated that top models likely exceed the previously assumed cap of 1 trillion parameters, with predictions suggesting figures as high as 3-6 trillion for future models like Grok 5. This investigation into the predictive power of various benchmarks, including MMLU Pro and the Intelligence Index, revealed that the Omniscience Accuracy benchmark had the highest predictive correlation (R²=0.84). However, metrics like Tau² and GDPVal associated with task performance were not predictive, hinting at a potential divergence between model knowledge capacity and task execution capabilities. The significance of this analysis extends to the AI/ML community's understanding of LLM development, as it leverages performance benchmarks to project model sizes, putting a spotlight on the trend of scaling models for enhanced accuracy. While the predictions for parameters were found to be potentially unrealistic, using models like the Intelligence Index seemed to yield more plausible metrics. This research contributes valuable insights into the characteristics and future trajectories of LLMs, emphasizing the impact of knowledge metrics on performance, while acknowledging the ongoing challenge of balancing model size, accuracy, and computational feasibility in the evolving landscape of artificial intelligence.