RESEARCH · RESEARCH · #1102
Study identifies frequent cost-inefficient behaviors in coding agents and evaluates mitigations
arXiv:2609.30725v1 analyzes 1,200 trajectories from Claude Code and Mini-SWE-Agent on SWE-bench Verified, identifying three recurring cost-inefficient behaviors (subsumed retrieval, similar script generation, test re-execution) that affect 79–98% of tasks and can account for up to 22.75% of task cost. The paper evaluates three mitigations (structure-aware retrieval, agent-synthesized skills, developer-designed skills) over 10k held-out trajectories and reports that structure-aware retrieval can sometimes increase costs (up to 28.14%), agent-synthesized skills give limited, low-level gains, while developer-designed skills reduce costs by up to 41.73%, about twice the maximum gain from agent-synthesized skills.
KEY POINTS
- arXiv:2609.30725v1 analyzes 1,200 trajectories from Claude Code and Mini-SWE-Agent on SWE-bench Verified, identifying three recurring cost-inefficient behaviors (subsumed retrieval, similar script generation, test re-execution) that affect 79–98% of tasks and can account for up to 22.75% of task cost.
- The paper evaluates three mitigations (structure-aware retrieval, agent-synthesized skills, developer-designed skills) over 10k held-out trajectories and reports that structure-aware retrieval can sometimes increase costs (up to 28.14%), agent-synthesized skills give limited, low-level gains, while developer-designed skills reduce costs by up to 41.73%, about twice the maximum gain from agent-synthesized skills.
- Quantifying and mitigating recurring cost-inefficient behaviors in coding agents matters because these behaviors materially increase operational costs and developer-designed skills appear to offer notably larger cost reductions for deployed agent workflows.
WHY IT MATTERS
Quantifying and mitigating recurring cost-inefficient behaviors in coding agents matters because these behaviors materially increase operational costs and developer-designed skills appear to offer notably larger cost reductions for deployed agent workflows.