RESEARCH · RESEARCH · #2981
ReplayRCA: intervention-based benchmark for diagnosing and overseeing LM agent failures
Researchers introduce ReplayRCA, a benchmark that evaluates post-hoc diagnosis and single-intervention live oversight of language-model agents by replaying a counterfactual action at a chosen step and measuring causal repair (continuation success minus a placebo replay). Using OpsWorld tasks, the paper reports that Grok 4.7 achieved the top causal repair (+51.3) on 62 calibrated failures, the top eight models were statistically indistinguishable, five failures were unrepaired by any model, forcing last-step interventions reduced performance by 17.1 points, and post-hoc diagnostic rank did not reliably predict live-supervision effectiveness; the environment and code are released on GitHub.
KEY POINTS
- Researchers introduce ReplayRCA, a benchmark that evaluates post-hoc diagnosis and single-intervention live oversight of language-model agents by replaying a counterfactual action at a chosen step and measuring causal repair (continuation success minus a placebo replay).
- Using OpsWorld tasks, the paper reports that Grok 4.7 achieved the top causal repair (+51.3) on 62 calibrated failures, the top eight models were statistically indistinguishable, five failures were unrepaired by any model, forcing last-step interventions reduced performance by 17.1 points, and post-hoc diagnostic rank did not reliably predict live-supervision effectiveness; the environment and code are released on GitHub.
- ReplayRCA provides an intervention-based, causally grounded evaluation that reveals important gaps between annotation-based attribution, post-hoc diagnosis, and actual live oversight performance for LM agents.
WHY IT MATTERS
ReplayRCA provides an intervention-based, causally grounded evaluation that reveals important gaps between annotation-based attribution, post-hoc diagnosis, and actual live oversight performance for LM agents.