RESEARCH · RESEARCH · #2183
Paper introduces Harness-Delta Attribution (HDA) to dissect gains from harness evolution
arXiv:2610.22466v1 introduces Harness-Delta Attribution (HDA), an evaluation protocol that decomposes performance gains from harness evolution into artifact-driven overfitting, test-time scaling, and a residual upper bound on potentially generalizable improvement. Applied to 11 executor models and three harness-evolution frameworks across math, coding, creativity, and agentic tasks, the study finds that overfitting and test-time scaling explain a large share of search-set gains (mean 54% and 14%, respectively), that gains often fail to transfer to held-out tasks, and that validation signals alone are insufficient to prevent overfitting.
KEY POINTS
- arXiv:2610.22466v1 introduces Harness-Delta Attribution (HDA), an evaluation protocol that decomposes performance gains from harness evolution into artifact-driven overfitting, test-time scaling, and a residual upper bound on potentially generalizable improvement.
- Applied to 11 executor models and three harness-evolution frameworks across math, coding, creativity, and agentic tasks, the study finds that overfitting and test-time scaling explain a large share of search-set gains (mean 54% and 14%, respectively), that gains often fail to transfer to held-out tasks, and that validation signals alone are insufficient to prevent overfitting.
- Because harness evolution is used to improve agentic systems without changing model weights, HDA reveals that much reported improvement may be overfitting or due to extra inference compute, so attribution-aware evaluation is needed.
WHY IT MATTERS
Because harness evolution is used to improve agentic systems without changing model weights, HDA reveals that much reported improvement may be overfitting or due to extra inference compute, so attribution-aware evaluation is needed.