ArXiv preprint 'Between Spoilers and Guesswork' on over-/under-specification in coding benchmarks
The arXiv preprint 'Between Spoilers and Guesswork' (arXiv:2610.22760v1) analyzes how task under-specification (causing guesswork) and over-specification (causing spoilers) harm repository-level coding benchmarks. The authors experiment with different specification levels, show that leaking ground-truth hints can introduce over-specification in benchmarks like SWE-bench Pro, FEA-bench, Doc2Feat-bench and Multi-SWE-bench, and propose techniques for producing parsimonious hints which they plan to release.