Tech Meridian ← LIVE FEED
PROMY MERIDIAN RU

RESEARCH · RESEARCH · #3627

OCTAVE: capability-resolved benchmark for end-to-end HTML5 game generation (arXiv:2611.17681v1)

OCTAVE is a capability-resolved benchmark for end-to-end HTML5 game generation organized into eight Capability Dimensions, 180 tasks, and 1,800 Evaluation Requirements drawn from player requests, commercial mechanics reconstruction, and expert design; it covers PC Web, Mobile Web, 2D and 3D. The paper reports static source-code requirement-level evaluation across 11 model-harness combinations (33.2% Completeness, 32.5% Richness), finds many static gate failures and missing artifacts, and shows no evaluated system exceeds 50% on both public and hidden requirements when excluding same-model self-evaluation.

KEY POINTS

  1. OCTAVE is a capability-resolved benchmark for end-to-end HTML5 game generation organized into eight Capability Dimensions, 180 tasks, and 1,800 Evaluation Requirements drawn from player requests, commercial mechanics reconstruction, and expert design; it covers PC Web, Mobile Web, 2D and 3D.
  2. The paper reports static source-code requirement-level evaluation across 11 model-harness combinations (33.2% Completeness, 32.5% Richness), finds many static gate failures and missing artifacts, and shows no evaluated system exceeds 50% on both public and hidden requirements when excluding same-model self-evaluation.
  3. OCTAVE provides a fine-grained, requirement-level diagnostic framework for whole-game generation that exposes specific capability gaps current systems still struggle with.

WHY IT MATTERS

OCTAVE provides a fine-grained, requirement-level diagnostic framework for whole-game generation that exposes specific capability gaps current systems still struggle with.

SOURCES & TIMELINE

1