RESEARCH · RESEARCH · #2321
ReuF2F: Lean 4 benchmark of 244 REU open math problems for coding agents
ReuF2F is a new Lean 4 benchmark (arXiv:2610.27046v1) of 244 formalized open problems sourced from Research Experiences for Undergraduates (REU). The benchmark packages each problem and its negation in a Lean project inside a Docker container, provides a unified evaluation harness (Leanimum-agent built on mini-SWE-agent), and reports that the best model in their evaluation solves only ~40% of the problems.
KEY POINTS
- ReuF2F is a new Lean 4 benchmark (arXiv:2610.27046v1) of 244 formalized open problems sourced from Research Experiences for Undergraduates (REU).
- The benchmark packages each problem and its negation in a Lean project inside a Docker container, provides a unified evaluation harness (Leanimum-agent built on mini-SWE-agent), and reports that the best model in their evaluation solves only ~40% of the problems.
- ReuF2F supplies a held-out, entry-level set of formalized open-research problems and a unified harness, enabling more realistic evaluation of coding agents on machine-checkable open problems and revealing substantial room for improvement.
WHY IT MATTERS
ReuF2F supplies a held-out, entry-level set of formalized open-research problems and a unified harness, enabling more realistic evaluation of coding agents on machine-checkable open problems and revealing substantial room for improvement.