RESEARCH · RESEARCH · #2517
Study finds agent 'harnesses' drive token costs but rarely improve coding performance
Researchers at Shandong University (Liu and Han) ran five models through three shipped coding-agent harnesses (Claude Code, OpenCode, mini-SWE-agent) on SWE-bench Verified and a larger 447-task pool, and found that swapping harnesses changes results little beyond run-to-run noise while token costs can triple because of large system prompts and repeated context. OpenCode consistently trailed the others on the larger pool (partly due to output limits), and many reported leaderboard gaps fall below statistical detectability given sample sizes and noise.
KEY POINTS
- Researchers at Shandong University (Liu and Han) ran five models through three shipped coding-agent harnesses (Claude Code, OpenCode, mini-SWE-agent) on SWE-bench Verified and a larger 447-task pool, and found that swapping harnesses changes results little beyond run-to-run noise while token costs can triple because of large system prompts and repeated context.
- OpenCode consistently trailed the others on the larger pool (partly due to output limits), and many reported leaderboard gaps fall below statistical detectability given sample sizes and noise.
- This matters because it challenges the practice of comparing agent stacks by leaderboard scores without controlling for run noise and token-cost differences, and highlights a significant hidden cost driver (system prompt size) in agent deployments.
WHY IT MATTERS
This matters because it challenges the practice of comparing agent stacks by leaderboard scores without controlling for run noise and token-cost differences, and highlights a significant hidden cost driver (system prompt size) in agent deployments.