RESEARCH · RESEARCH · #2715
Counting the Bandages: On measuring the time that AI saves
New arXiv paper (2610.35942v1) argues for a 'ledger' view of AI time-savings that pairs measured time saved with counterfactual excess escaped defects, defines model-free quantities (time saved, excess escaped defects) and a break-even defect-cost decision number, presents a baseline checking model, analyzes identifiability in randomized studies, replicates METR's developer study (same 246 issues show a 19%–24% slowdown depending on estimand; 34% without imbalance adjustment; AI helped on 3%–82% of issues), reports that GDPval's review-every-output workflow caps GPT-5 speedup at 3.6x, and tests model predictions in a small pre-specified experiment where reduced checking (informed by optimistic drafter error rates) produced faster but worse outputs.
KEY POINTS
- New arXiv paper (2610.35942v1) argues for a 'ledger' view of AI time-savings that pairs measured time saved with counterfactual excess escaped defects, defines model-free quantities (time saved, excess escaped defects) and a break-even defect-cost decision number, presents a baseline checking model, analyzes identifiability in randomized studies, replicates METR's developer study (same 246 issues show a 19%–24% slowdown depending on estimand; 34% without imbalance adjustment; AI helped on 3%–82% of issues), reports that GDPval's review-every-output workflow caps GPT-5 speedup at 3.6x, and tests model predictions in a small pre-specified experiment where reduced checking (informed by optimistic drafter error rates) produced faster but worse outputs.
- It gives a concrete, testable framework and empirical bounds for evaluating AI productivity gains alongside the trade-off in escaped defects, which is critical for responsible deployment and ROI calculations.
- Counting the Bandages: On measuring the time that AI saves
WHY IT MATTERS
It gives a concrete, testable framework and empirical bounds for evaluating AI productivity gains alongside the trade-off in escaped defects, which is critical for responsible deployment and ROI calculations.