Tech Meridian ← LIVE FEED
PROMY MERIDIAN RU

RESEARCH · RESEARCH · #3110

ViBench 2.0: benchmark for long-horizon software building on Replit

ViBench 2.0 (arXiv:2611.05620v1) is a new benchmark for long-horizon software building on Replit’s agentic engineering platform. It models 17 full‑stack web products that grow over 241 feature requests, uses a browser agent to run 544 test plans with a validated grading protocol, and reports evaluations of nine models that reveal a reliability gap (individual test-plan pass rates up to 98% but fully working builds at most 65%); eight products and the test harness/grader are released, nine are held out.

KEY POINTS

  1. ViBench 2.0 (arXiv:2611.05620v1) is a new benchmark for long-horizon software building on Replit’s agentic engineering platform.
  2. It models 17 full‑stack web products that grow over 241 feature requests, uses a browser agent to run 544 test plans with a validated grading protocol, and reports evaluations of nine models that reveal a reliability gap (individual test-plan pass rates up to 98% but fully working builds at most 65%); eight products and the test harness/grader are released, nine are held out.
  3. ViBench 2.0 matters because it evaluates end‑to‑end, long‑horizon product construction and exposes a gap between per-test reliability and delivering fully working builds, informing model selection for production agentic coding systems.

WHY IT MATTERS

ViBench 2.0 matters because it evaluates end‑to‑end, long‑horizon product construction and exposes a gap between per-test reliability and delivering fully working builds, informing model selection for production agentic coding systems.

SOURCES & TIMELINE

1