RESEARCH · RESEARCH · #3572
Termination lag in coding agents (arXiv:2611.14960v1)
The paper (arXiv:2611.14960v1) defines and measures “termination lag” — steps between an agent's first correct patch and the end of its run — by recording and testing every distinct repository patch across five models, two agent frameworks, and three SWE-bench datasets. It finds 38–82% of successful runs produce a correct solution on the initial edit, yet agents often continue working: median post-solution steps account for 35–37% for Qwen3-Coder and 54–60% for Laguna-XS.2 on SWE-bench Verified and Multilingual, and in 1–8% of cases agents later undo a previously correct patch.
KEY POINTS
- The paper (arXiv:2611.14960v1) defines and measures “termination lag” — steps between an agent's first correct patch and the end of its run — by recording and testing every distinct repository patch across five models, two agent frameworks, and three SWE-bench datasets.
- It finds 38–82% of successful runs produce a correct solution on the initial edit, yet agents often continue working: median post-solution steps account for 35–37% for Qwen3-Coder and 54–60% for Laguna-XS.2 on SWE-bench Verified and Multilingual, and in 1–8% of cases agents later undo a previously correct patch.
- This quantifies a common inefficiency and occasional regression risk in code-writing agents, affecting evaluation, compute cost, and reliability.
WHY IT MATTERS
This quantifies a common inefficiency and occasional regression risk in code-writing agents, affecting evaluation, compute cost, and reliability.