RESEARCH · RESEARCH · #1832
Vals AI: agent teams cost far more for minimal quality gains
Evals firm Vals AI benchmarked GPT-6 Sol and Claude Opus 5.5 on the Vibe Code Bench as solo agents and in multi-agent teams at medium and maximum reasoning effort. Teams consumed 1.8×–5.1× more tokens but showed almost no consistent quality improvement—only GPT-6 Sol at medium reasoning produced a statistically significant +7.3 score—while at maximum reasoning teams offered no real advantage; Anthropic, ProgramBench tests, and OpenAI researchers similarly report that agent swarms mainly speed up work at higher cost and face coordination limits.
KEY POINTS
- Evals firm Vals AI benchmarked GPT-6 Sol and Claude Opus 5.5 on the Vibe Code Bench as solo agents and in multi-agent teams at medium and maximum reasoning effort.
- Teams consumed 1.8×–5.1× more tokens but showed almost no consistent quality improvement—only GPT-6 Sol at medium reasoning produced a statistically significant +7.3 score—while at maximum reasoning teams offered no real advantage; Anthropic, ProgramBench tests, and OpenAI researchers similarly report that agent swarms mainly speed up work at higher cost and face coordination limits.
- This matters because it suggests multi-agent systems are often not cost-effective—typically buying speed rather than meaningful quality gains—affecting deployment and cost decisions for teams using agent swarms.
WHY IT MATTERS
This matters because it suggests multi-agent systems are often not cost-effective—typically buying speed rather than meaningful quality gains—affecting deployment and cost decisions for teams using agent swarms.