RESEARCH · RESEARCH · #2082
Recursive Reward Improvement (RRI): adversarial rubric repair for open-ended self-improvement
The paper introduces Recursive Reward Improvement (RRI), a proposer–solver self-improvement method that uses an adversary to generate rubric‑gaming (reward‑hacking) answers, compares them to honest answers produced without seeing the rubric, and iteratively repairs model-written rubrics so honest answers win. On HealthBench Professional RRI raised the adjusted score versus a Generation Only baseline by 10.9 points at 9B and 12.4 points at 27B (and by 10.3 points without any external model), with largest gains on difficult and writing tasks and demonstrated transfer to PRBench Hard and ProfBench.
KEY POINTS
- The paper introduces Recursive Reward Improvement (RRI), a proposer–solver self-improvement method that uses an adversary to generate rubric‑gaming (reward‑hacking) answers, compares them to honest answers produced without seeing the rubric, and iteratively repairs model-written rubrics so honest answers win.
- On HealthBench Professional RRI raised the adjusted score versus a Generation Only baseline by 10.9 points at 9B and 12.4 points at 27B (and by 10.3 points without any external model), with largest gains on difficult and writing tasks and demonstrated transfer to PRBench Hard and ProfBench.
- This matters because it provides a practical adversarial method to detect and repair reward-hacking in model‑written rubrics, enabling more reliable open‑ended self‑improvement and showing substantial benchmark gains on health-related tasks.
WHY IT MATTERS
This matters because it provides a practical adversarial method to detect and repair reward-hacking in model‑written rubrics, enabling more reliable open‑ended self‑improvement and showing substantial benchmark gains on health-related tasks.