RESEARCH · RESEARCH · #2472
RegretBench and BackUpper: measuring and automating backups for coding agents
An arXiv paper (arXiv:2610.28934v1) introduces RegretBench, a 205-task benchmark of real-world "regret" moments where users wished they'd preserved state, and BackUpper, a harness plug-in that checks tool calls and preserves affected state before modifications. Across six frontier models agents complete 86.1% of tasks but back up before regret moments only 15.2%; BackUpper covers all regret actions, restores states with 92.2% average similarity, costs about $0.014 per task, uses 7× fewer tokens than a self-backup baseline, and can match user backup preferences with an 84.5% F1 score.
KEY POINTS
- An arXiv paper (arXiv:2610.28934v1) introduces RegretBench, a 205-task benchmark of real-world "regret" moments where users wished they'd preserved state, and BackUpper, a harness plug-in that checks tool calls and preserves affected state before modifications.
- Across six frontier models agents complete 86.1% of tasks but back up before regret moments only 15.2%; BackUpper covers all regret actions, restores states with 92.2% average similarity, costs about $0.014 per task, uses 7× fewer tokens than a self-backup baseline, and can match user backup preferences with an 84.5% F1 score.
- Automated backups make coding agents' changes reversible and safer at low cost, addressing an overlooked practical failure mode when agents modify real-world state.
WHY IT MATTERS
Automated backups make coding agents' changes reversible and safer at low cost, addressing an overlooked practical failure mode when agents modify real-world state.