RESEARCH · RESEARCH · #711
GameASG-Bench: benchmark for behavioral testability in autonomous game software generation (arXiv:2609.21293v1)
The paper introduces GameASG-Bench, a benchmark that requires behavioral testability for autonomous software generation in game development by specifying an evaluation interface and providing static L1 source checks plus browser-executed L2 runtime checks. The release contains 47 browser-native game-generation tasks across 12 genres with reference implementations; experiments on nine agent stacks report up to 93.2% mean L2 check pass rate but only a 55.3% strict task success rate (26/47), and show gains for DeepSeek-V4-Flash with full tool access and larger nominal turn budgets.
KEY POINTS
- The paper introduces GameASG-Bench, a benchmark that requires behavioral testability for autonomous software generation in game development by specifying an evaluation interface and providing static L1 source checks plus browser-executed L2 runtime checks.
- The release contains 47 browser-native game-generation tasks across 12 genres with reference implementations; experiments on nine agent stacks report up to 93.2% mean L2 check pass rate but only a 55.3% strict task success rate (26/47), and show gains for DeepSeek-V4-Flash with full tool access and larger nominal turn budgets.
- GameASG-Bench matters because it makes behavioral testability a first-class requirement for end-to-end code-generating agents and exposes task-level compliance gaps that average check rates can hide.
WHY IT MATTERS
GameASG-Bench matters because it makes behavioral testability a first-class requirement for end-to-end code-generating agents and exposes task-level compliance gaps that average check rates can hide.