Recursive Reward Improvement (RRI): adversarial rubric repair for open-ended self-improvement
The paper introduces Recursive Reward Improvement (RRI), a proposer–solver self-improvement method that uses an adversary to generate rubric‑gaming (reward‑hacking) answers, compares them to honest answers produced without seeing the rubric, and iteratively repairs model-written rubrics so honest answers win. On HealthBench Professional RRI raised the adjusted score versus a Generation Only baseline by 10.9 points at 9B and 12.4 points at 27B (and by 10.3 points without any external model), with largest gains on difficult and writing tasks and demonstrated transfer to PRBench Hard and ProfBench.