RSHD: Recovering supervision from failed rollouts to improve GRPO
The paper (arXiv:2610.21018v1) introduces Replay-Selected Hindsight Distillation (RSHD), a method that turns failed rollouts into supervision by having the policy write corrective notes, re-solving problems with those notes, admitting notes whose replays succeed more often than the original group, and distilling admitted replays into the base policy. RSHD requires training only the policy and no extra inference at deployment; across model families/sizes and two training domains it consistently improves GRPO on in-domain benchmarks and yields gains on out-of-domain evaluations.