Tech Meridian ← LIVE FEED
PROMY MERIDIAN RU

RESEARCH · RESEARCH · #3264

Co-Design Bench finds coding agents fully complete only ~26% of real user tasks

Researchers (led by Alibaba and several Asian universities) released Co-Design Bench, a new benchmark using real user change requests for web apps; across ten models the best system (GPT-6 Astra) met about 70% of requirements on average but produced a fully correct result in only 26.4% of tasks, meaning human cleanup is still needed on nearly three out of four tasks. The study also shows performance depends strongly on task complexity and the agent harness (execution environment), so authors recommend reporting model and environment together.

KEY POINTS

  1. Researchers (led by Alibaba and several Asian universities) released Co-Design Bench, a new benchmark using real user change requests for web apps; across ten models the best system (GPT-6 Astra) met about 70% of requirements on average but produced a fully correct result in only 26.4% of tasks, meaning human cleanup is still needed on nearly three out of four tasks.
  2. The study also shows performance depends strongly on task complexity and the agent harness (execution environment), so authors recommend reporting model and environment together.
  3. This benchmark highlights a large gap between near‑perfect scores on synthetic tests and real‑world usability of coding agents, affecting how companies should evaluate and deploy them.

WHY IT MATTERS

This benchmark highlights a large gap between near‑perfect scores on synthetic tests and real‑world usability of coding agents, affecting how companies should evaluate and deploy them.

SOURCES & TIMELINE

1