Tech Meridian ← ENTITY INDEX
PROMY MERIDIAN RU

TOPIC · ENTITY #27772

ReplayRCA (benchmark)

Related event timeline, sources and context from the news index.

EVENT TIMELINE

1

RESEARCH · 1 SOURCE · arXiv cs.AI

ReplayRCA: intervention-based benchmark for diagnosing and overseeing LM agent failures

Researchers introduce ReplayRCA, a benchmark that evaluates post-hoc diagnosis and single-intervention live oversight of language-model agents by replaying a counterfactual action at a chosen step and measuring causal repair (continuation success minus a placebo replay). Using OpsWorld tasks, the paper reports that Grok 4.7 achieved the top causal repair (+51.3) on 62 calibrated failures, the top eight models were statistically indistinguishable, five failures were unrepaired by any model, forcing last-step interventions reduced performance by 17.1 points, and post-hoc diagnostic rank did not reliably predict live-supervision effectiveness; the environment and code are released on GitHub.

7.0