EdgeBench: Unveiling scaling laws of (AI) learning from real-world environments (github.com)

EdgeBench has launched a novel benchmark comprising 134 real-world tasks designed to evaluate how autonomous AI agents learn and adapt over extended periods. Unlike traditional evaluations that focus on one-off performances, EdgeBench allows agents to engage in realistic environments for over 12 hours per task, enabling the tracking of incremental improvements rather than just final results. This significant methodology shift aims to enhance our understanding of AI learning dynamics by providing detailed insights into agent behavior across various tasks. The benchmark's analysis, generated from approximately 38,000 hours of agent interaction, reveals that the performance of AI models follows a log-sigmoid scaling law in relation to interaction time. This finding is critical for the AI and machine learning communities, as it underscores the potential for foundational advancements in the iterative training process of AI. Key AI models, including Claude Opus 4.8 and GPT-5.5, were evaluated, with results highlighting their varied efficacy across specialized task categories such as Scientific & ML, Systems & SE, and Optimization. The public release of this framework and its tasks not only fosters transparency but also encourages broader experimentation within the field.