LLMs tested on historical prediction markets, NONE outperformed the market (arxiv.org)

Researchers have introduced Forecast-Dojo, a new benchmarking environment designed to assess and train large language model (LLM) agents on forecasting tasks using historical prediction market data. This innovative platform allows for the replay of approximately 1,568 resolved market events and leverages 18.8 million dated news articles, enabling reproducible evaluations and longitudinal analyses. The study revealed that while certain research tools improved forecasting performance as agents learned to incorporate new evidence, none of the twelve tested LLMs outperformed predictions made by the historical market. The significance of this development lies in its potential to advance the capabilities of LLMs in dynamic reasoning and evidence assimilation, critical aspects of real-world forecasting. Despite the challenges faced, including the inherent complexity of adapting to ever-evolving data, Forecast-Dojo shows promise in enhancing LLMs through structured training. The notable increment in accuracy from 34.8% to 42.8% in fine-tuned models illustrates the environment’s value, particularly in simulating real-world decision-making processes, although it also highlights the continuing superiority of established market predictions over AI approaches in this domain.