RESEARCH · RESEARCH · #1965
Study: delegating exploration and planning lets agent teams reach near-frontier coding performance at far lower cost
A controlled study of coding-agent teams finds that well-organized multi-agent systems can outperform single agents on ProgramBench by up to ~10 percentage points but typically cost 3–7× more. Crucially, delegating only exploration and planning can match the performance of delegating planning, implementation, and verification together at roughly half the cost — e.g., a Sol lead with 16 inexpensive Luna planning workers reached 61.8% reward vs 63.2% for an all‑Sol team while cutting API cost from $510 to $155; the paper also situates these results amid recent large multi-agent coding projects (Claude, Cursor, Bun).
KEY POINTS
- A controlled study of coding-agent teams finds that well-organized multi-agent systems can outperform single agents on ProgramBench by up to ~10 percentage points but typically cost 3–7× more.
- Crucially, delegating only exploration and planning can match the performance of delegating planning, implementation, and verification together at roughly half the cost — e.g., a Sol lead with 16 inexpensive Luna planning workers reached 61.8% reward vs 63.2% for an all‑Sol team while cutting API cost from $510 to $155; the paper also situates these results amid recent large multi-agent coding projects (Claude, Cursor, Bun).
- It identifies where inference compute in agent teams is most cost-effective, giving actionable design guidance to achieve near-frontier software performance at much lower API/inference cost.
WHY IT MATTERS
It identifies where inference compute in agent teams is most cost-effective, giving actionable design guidance to achieve near-frontier software performance at much lower API/inference cost.