RESEARCH · RESEARCH · #3110
ViBench 2.0: benchmark for long-horizon software building on Replit
ViBench 2.0 (arXiv:2611.05620v1) is a new benchmark for long-horizon software building on Replit’s agentic engineering platform. It models 17 full‑stack web products that grow over 241 feature requests, uses a browser agent to run 544 test plans with a validated grading protocol, and reports evaluations of nine models that reveal a reliability gap (individual test-plan pass rates up to 98% but fully working builds at most 65%); eight products and the test harness/grader are released, nine are held out.
KEY POINTS
- ViBench 2.0 (arXiv:2611.05620v1) is a new benchmark for long-horizon software building on Replit’s agentic engineering platform.
- It models 17 full‑stack web products that grow over 241 feature requests, uses a browser agent to run 544 test plans with a validated grading protocol, and reports evaluations of nine models that reveal a reliability gap (individual test-plan pass rates up to 98% but fully working builds at most 65%); eight products and the test harness/grader are released, nine are held out.
- ViBench 2.0 matters because it evaluates end‑to‑end, long‑horizon product construction and exposes a gap between per-test reliability and delivering fully working builds, informing model selection for production agentic coding systems.
WHY IT MATTERS
ViBench 2.0 matters because it evaluates end‑to‑end, long‑horizon product construction and exposes a gap between per-test reliability and delivering fully working builds, informing model selection for production agentic coding systems.