Benchy: a semantic language and execution engine for task-oriented AI benchmarks (arXiv:2609.30550v1)
The paper introduces Benchy, a semantic specification language and execution engine that encodes benchmarks as canonical YAML programs (B=(P,S,D)), deterministically compiles them to a JSON IR, and exposes a universal runtime contract (named-field input/output objects) so external AI systems adapt only at the boundary. It defines the semantic object model, ontology, validation and scoring/failure semantics, compilation/execution architecture, and includes an appendix that fixes the engineering contract for a first engine implementation.