RESEARCH · RESEARCH · #1356
Paper: Minimal-harness coding agents match complex harnesses for autonomous ML engineering
The authors evaluate autonomous machine-learning-engineering (MLE) agents and report that, given the same time budget and the same frontier LLM backbone, open-source state-of-the-art multi-agent harnesses offer no advantage over a single-session minimal-harness coding agent baseline. Large-scale ablation studies indicate the extra orchestration layers are redundant in the coding-agent setting, implying the backbone model is the primary driver of current leaderboard performance.
KEY POINTS
- The authors evaluate autonomous machine-learning-engineering (MLE) agents and report that, given the same time budget and the same frontier LLM backbone, open-source state-of-the-art multi-agent harnesses offer no advantage over a single-session minimal-harness coding agent baseline.
- Large-scale ablation studies indicate the extra orchestration layers are redundant in the coding-agent setting, implying the backbone model is the primary driver of current leaderboard performance.
- This finding challenges the value of increasingly elaborate harnesses and suggests research and engineering effort may be better spent on improving the LLM backbone and coding-agent primitives for current MLE benchmarks.
WHY IT MATTERS
This finding challenges the value of increasingly elaborate harnesses and suggests research and engineering effort may be better spent on improving the LLM backbone and coding-agent primitives for current MLE benchmarks.