RESEARCH · RESEARCH · #1764
OpenProblemBench: benchmark of 82 open math and theoretical-physics problems for AI
OpenProblemBench is a new benchmark (arXiv:2610.11118v1) comprising 82 unresolved problems from mathematics and theoretical physics, each provided with context, assumptions, and prior progress to support investigation. Four evaluator models judged submissions without reference solutions; across seven configurations GPT-6-Astra achieved a mean judged solve rate of 14.0%, versus 5.5–6.7% for evaluated full-size open models and 2.4–3.7% for Flash models.
KEY POINTS
- OpenProblemBench is a new benchmark (arXiv:2610.11118v1) comprising 82 unresolved problems from mathematics and theoretical physics, each provided with context, assumptions, and prior progress to support investigation.
- Four evaluator models judged submissions without reference solutions; across seven configurations GPT-6-Astra achieved a mean judged solve rate of 14.0%, versus 5.5–6.7% for evaluated full-size open models and 2.4–3.7% for Flash models.
- By framing evaluation around unresolved research questions with clear correctness checks, OpenProblemBench creates a measurable testbed for assessing AI’s ability to contribute to foundational theoretical science.
WHY IT MATTERS
By framing evaluation around unresolved research questions with clear correctness checks, OpenProblemBench creates a measurable testbed for assessing AI’s ability to contribute to foundational theoretical science.