RESEARCH · RESEARCH · #2051
TheoPhysEval benchmark: GPT-6-Astra solves 49 of 50 theoretical physics problems
Thirty-five researchers released TheoPhysEval, a 50-problem benchmark of unpublished theoretical physics calculations with verifiable answers. GPT-6-Astra (running in OpenAI's Codex environment) solved 49 of 50 problems (98%) at an average cost under $3 per problem; Anthropic's Opus 5.5 scored 92%, Gemini 3.8 Flash 68%, and several open-weight models scored lower. Tests were run with and without web access and authors note models often performed well without internet, implying substantial memorization of the literature; the dataset and test code are available on Hugging Face.
KEY POINTS
- Thirty-five researchers released TheoPhysEval, a 50-problem benchmark of unpublished theoretical physics calculations with verifiable answers.
- GPT-6-Astra (running in OpenAI's Codex environment) solved 49 of 50 problems (98%) at an average cost under $3 per problem; Anthropic's Opus 5.5 scored 92%, Gemini 3.8 Flash 68%, and several open-weight models scored lower.
- Tests were run with and without web access and authors note models often performed well without internet, implying substantial memorization of the literature; the dataset and test code are available on Hugging Face.
WHY IT MATTERS
The result suggests leading language models can already outperform many human specialists on concrete, verifiable theoretical calculations, shifting the boundary of where AI can automate research workloads.