RESEARCH · RESEARCH · #2483
Agent4Sci testbed (arXiv:2610.28763v1) evaluates frontier coding agents on 20 ML‑for‑science benchmarks
The paper introduces Agent4Sci, a long‑horizon testbed that measures how well current coding agents can improve published best models across 20 scientific ML benchmarks in 11 domains. Empirically, frontier agents (e.g., Claude Opus) improve published baselines (Claude Opus: 10.4% mean relative improvement), but gains mainly come from ML‑engineering tricks; simple plug‑in skills for ideation and experiment management boost Claude Opus to 20.1% and Codex to 8.1%. Code and data are available on GitHub and Hugging Face.
KEY POINTS
- The paper introduces Agent4Sci, a long‑horizon testbed that measures how well current coding agents can improve published best models across 20 scientific ML benchmarks in 11 domains.
- Empirically, frontier agents (e.g., Claude Opus) improve published baselines (Claude Opus: 10.4% mean relative improvement), but gains mainly come from ML‑engineering tricks; simple plug‑in skills for ideation and experiment management boost Claude Opus to 20.1% and Codex to 8.1%.
- Code and data are available on GitHub and Hugging Face.
WHY IT MATTERS
Agent4Sci provides a reproducible, multi‑domain benchmark and dataset that reveals where frontier agents succeed or fail at research‑style ML tasks and which lightweight interventions shift them from engineering hacks toward more domain‑relevant insight.