NeurIPS 2025 E2LM Competition: Early Training Evaluation of Language Models (huggingface.co)

The NeurIPS 2025 E2LM Competition has been announced, focusing on the early training evaluation of Large Language Models (LLMs) specifically within the scientific knowledge domain. This initiative aims to develop new benchmarks that can accurately capture critical signals during the early stages of LLM training, where traditional evaluation metrics often fall short. Participants will utilize the lm-evaluation-harness library through a Hugging Face Space, with an active leaderboard to encourage collaboration and competition. The competition's unique scoring system evaluates submissions based on three key metrics: signal quality, ranking consistency, and compliance with established scientific knowledge. Significantly, the competition emphasizes the importance of signal quality and adherence to scientific accuracy, weighing these metrics more heavily in the final score. This approach not only pushes for better methodological practices in early LLM training but also addresses ongoing challenges of information leakage and biased assessments that can arise from current evaluation standards. By enabling participants to compute certain metrics locally with smaller model checkpoints while maintaining the confidentiality of larger models, the contest encourages innovative solutions that can genuinely advance the evaluation landscape for LLMs. The competition kicks off on July 14, 2025, with significant prizes awarded, including a total of $12,000 for the top three placements and student awards.