Paper introduces Harness-Delta Attribution (HDA) to dissect gains from harness evolution
arXiv:2610.22466v1 introduces Harness-Delta Attribution (HDA), an evaluation protocol that decomposes performance gains from harness evolution into artifact-driven overfitting, test-time scaling, and a residual upper bound on potentially generalizable improvement. Applied to 11 executor models and three harness-evolution frameworks across math, coding, creativity, and agentic tasks, the study finds that overfitting and test-time scaling explain a large share of search-set gains (mean 54% and 14%, respectively), that gains often fail to transfer to held-out tasks, and that validation signals alone are insufficient to prevent overfitting.