RESEARCH · RESEARCH · #2083
RSHD: Recovering supervision from failed rollouts to improve GRPO
The paper (arXiv:2610.21018v1) introduces Replay-Selected Hindsight Distillation (RSHD), a method that turns failed rollouts into supervision by having the policy write corrective notes, re-solving problems with those notes, admitting notes whose replays succeed more often than the original group, and distilling admitted replays into the base policy. RSHD requires training only the policy and no extra inference at deployment; across model families/sizes and two training domains it consistently improves GRPO on in-domain benchmarks and yields gains on out-of-domain evaluations.
KEY POINTS
- The paper (arXiv:2610.21018v1) introduces Replay-Selected Hindsight Distillation (RSHD), a method that turns failed rollouts into supervision by having the policy write corrective notes, re-solving problems with those notes, admitting notes whose replays succeed more often than the original group, and distilling admitted replays into the base policy.
- RSHD requires training only the policy and no extra inference at deployment; across model families/sizes and two training domains it consistently improves GRPO on in-domain benchmarks and yields gains on out-of-domain evaluations.
- RSHD matters because it recovers usable supervision even when every rollout in a group fails, enabling learning from failures and improving generalization without changing deployment requirements.
WHY IT MATTERS
RSHD matters because it recovers usable supervision even when every rollout in a group fails, enabling learning from failures and improving generalization without changing deployment requirements.