Futuresearch Evals (evals.futuresearch.ai)

Futuresearch has announced the release of Bench to the Future 3 (BTF-3), their latest benchmark for evaluating forecasting and research agents, featuring 2,386 resolved questions within a fixed web corpus. BTF-3 includes 2,015 binary questions and 371 numeric ones, with results assessed using Brier scores; notably, the FutureSearch SOTA emerged as the leader with a pooled Brier score of 0.116. This benchmark serves as a critical tool for gauging advancements in AI models focused on forecasting and decision-making, revealing how different agents perform across various question types and their effectiveness at integrating recent knowledge. The significance of BTF-3 for the AI and ML community lies in its potential to refine the development of forecasting agents by providing a comprehensive evaluation framework. The results emphasize the importance of training cutoffs, with the latest models like Claude Opus 5 demonstrating heightened accuracy due to fresher knowledge of world events. Additionally, BTF-3 underscores the statistical methodologies employed to ensure reliable comparisons between models, including bootstrapping techniques for confidence intervals. This release not only enhances the benchmarking landscape but also propels competition among leading AI agents, ultimately advancing their capabilities in real-world forecasting scenarios.