RESEARCH · RESEARCH · #2890
OpenFrontiers: dataset and benchmark for LLM scientific rediscovery (arXiv:2610.37739v1)
OpenFrontiers is a cross-disciplinary resource combining a structured corpus of 12,151 open research problems with a retrospective benchmark of 200 mathematics problems (110 resolved in 2026, 90 still open), each formalized as Lean 4 statements with automated checks and independent reviews; a higher-quality HQ-50 subset is provided. The authors publish an evaluation protocol (closed-book, frozen-retrieval, live-web) and report results evaluating seven LLM agents on HQ-50 under a standardized budget, finding closed-book rediscovery rare (at most 7 of 40 resolved problems) while live-web successes largely reflect retrieval of published solutions; the dataset is hosted at the linked Hugging Face collection.
KEY POINTS
- OpenFrontiers is a cross-disciplinary resource combining a structured corpus of 12,151 open research problems with a retrospective benchmark of 200 mathematics problems (110 resolved in 2026, 90 still open), each formalized as Lean 4 statements with automated checks and independent reviews; a higher-quality HQ-50 subset is provided.
- The authors publish an evaluation protocol (closed-book, frozen-retrieval, live-web) and report results evaluating seven LLM agents on HQ-50 under a standardized budget, finding closed-book rediscovery rare (at most 7 of 40 resolved problems) while live-web successes largely reflect retrieval of published solutions; the dataset is hosted at the linked Hugging Face collection.
- OpenFrontiers creates a standardized, formalized benchmark linking an open-problem corpus to graded proofs, enabling reproducible measurement of LLM progress on research-level problem solving and scientific rediscovery.
WHY IT MATTERS
OpenFrontiers creates a standardized, formalized benchmark linking an open-problem corpus to graded proofs, enabling reproducible measurement of LLM progress on research-level problem solving and scientific rediscovery.