RESEARCH · RESEARCH · #2172
GlobeBench benchmark evaluates text-based world model capabilities across 1,632 tasks
An arXiv paper (arXiv:2610.22708v1) introduces GlobeBench, a benchmark of 1,632 tasks spanning five domains and 19 simulation skills to evaluate text-based world models’ ability to predict environment responses from agent trajectories. The authors evaluate 20 models (including general-purpose LLMs and specialized text-world models), report GPT-6 Astra as the top performer with a 49.57% task success rate, find accuracy falls sharply when actions must propagate to dependent objects (average success 5.1%), and note 631 tasks remain unsolved by any model.
KEY POINTS
- An arXiv paper (arXiv:2610.22708v1) introduces GlobeBench, a benchmark of 1,632 tasks spanning five domains and 19 simulation skills to evaluate text-based world models’ ability to predict environment responses from agent trajectories.
- The authors evaluate 20 models (including general-purpose LLMs and specialized text-world models), report GPT-6 Astra as the top performer with a 49.57% task success rate, find accuracy falls sharply when actions must propagate to dependent objects (average success 5.1%), and note 631 tasks remain unsolved by any model.
- A standardized, large-scale benchmark reveals substantial gaps in LLMs’ world-modeling abilities and provides a shared evaluation to guide research and improvement.
WHY IT MATTERS
A standardized, large-scale benchmark reveals substantial gaps in LLMs’ world-modeling abilities and provides a shared evaluation to guide research and improvement.