How Accurate Have AI Progress Forecasts Been So Far? (forecastingresearch.substack.com)

The Forecasting Research Institute (FRI) has released insights on the accuracy of predictions regarding AI progress, revealing a tendency among experts to underestimate advancements significantly. Since mid-2022, FRI has tracked forecasts related to various aspects of AI, including capabilities, adoption, and societal impacts. The findings suggest forecasters have notably misjudged benchmark performance — often by years — while their predictions on AI adoption and broader economic effects remain variable with a tendency towards underestimation. For instance, predictions about key milestones, like AI achieving gold-level performance in mathematical benchmarks, were surpassed by up to a decade. This study is crucial for the AI/ML community as it highlights the disconnect between expert forecasts and actual development timelines, which can influence policy decisions and funding allocations. It underscores the need for more rigorous forecasting methods to better navigate the complexities of AI's future trajectory. Furthermore, the challenges faced in assessing forecasts — particularly in identifying overestimations versus underestimations — point to the necessity for continuous updates and refinements in forecasting practices within the rapidly evolving AI landscape. The next installment from FRI will focus on improving forecast accuracy, making this a pivotal discussion for stakeholders in the field.