RESEARCH · RESEARCH · #1009
Forecast-Dojo: replayable benchmark and training environment for LLM forecasting agents (arXiv:2609.28876v1)
Forecast-Dojo is a replayable environment that pairs 1,568 resolved Polymarket prediction questions with 18.8M dated news articles so agents can research events and revisit forecasts at successive historical dates. The paper evaluates 12 models, finds research tools reduce Brier score and that forecasts improve as events unfold (though all models still trail historical market forecasts), and provides interaction trajectories plus a supervised fine-tuning proof of concept.
KEY POINTS
- Forecast-Dojo is a replayable environment that pairs 1,568 resolved Polymarket prediction questions with 18.8M dated news articles so agents can research events and revisit forecasts at successive historical dates.
- The paper evaluates 12 models, finds research tools reduce Brier score and that forecasts improve as events unfold (though all models still trail historical market forecasts), and provides interaction trajectories plus a supervised fine-tuning proof of concept.
- It provides a large, replayable benchmark and training dataset that enables reproducible evaluation and learning for LLM forecasting agents without waiting for new events to resolve.
WHY IT MATTERS
It provides a large, replayable benchmark and training dataset that enables reproducible evaluation and learning for LLM forecasting agents without waiting for new events to resolve.