Study finds agent 'harnesses' drive token costs but rarely improve coding performance
Researchers at Shandong University (Liu and Han) ran five models through three shipped coding-agent harnesses (Claude Code, OpenCode, mini-SWE-agent) on SWE-bench Verified and a larger 447-task pool, and found that swapping harnesses changes results little beyond run-to-run noise while token costs can triple because of large system prompts and repeated context. OpenCode consistently trailed the others on the larger pool (partly due to output limits), and many reported leaderboard gaps fall below statistical detectability given sample sizes and noise.