Neo Scored 34.2% SOTA on OpenAI MLE-Bench (github.com)

The MLE-Bench benchmark, designed to rigorously evaluate AI agents on machine learning engineering challenges drawn from 75 Kaggle competitions, has seen a new state-of-the-art performance milestone. The Neo multi-agent system recently achieved a 34.2% overall score on MLE-Bench, outperforming notable contenders like R&D-Agent o3 combined with GPT-4.1 and ML-Master deepseek-r1. This benchmark measures agents' capabilities across low, medium, and high complexity tasks spanning diverse domains such as image classification, tabular data, and text analysis, making it a comprehensive gauge of AI systems’ practical ML problem-solving prowess. This advancement is significant for the AI/ML community as it demonstrates multi-agent systems' growing ability to automate complex end-to-end ML workflows, including dataset preparation, model training, and evaluation—facets critical to real-world deployment. The benchmark’s open-source code, evaluation logic, and grading tools foster transparency and reproducibility, enabling researchers to fairly compare new agents under standardized compute constraints (36 vCPUs, 24GB GPU, 440GB RAM, 24-hour runtime). The availability of a "lite" version focusing on low-complexity tasks offers a cost-effective entry point for benchmarking without sacrificing comparability. Technically, MLE-Bench also incorporates additional integrity checks like rule violation and plagiarism detectors and supports detailed metric breakdowns by task complexity, encouraging nuanced agent development. The dataset split and grading mechanism are carefully engineered to mitigate Kaggle’s held-out test set limitations, allowing independent evaluation. Neo’s leading score highlights the potential of coordinated multi-agent approaches in pushing the boundaries of ML engineering automation and underscores MLE-Bench’s role as a crucial benchmark for advancing AI-driven machine learning workflows.