RESEARCH · RESEARCH · #1762
Adaptive Reference Guidance (ARG) for prefix continuation in reasoning RL (arXiv:2610.11128v1)
The arXiv preprint (2610.11128v1) analyzes how much reference guidance to provide when training reasoning models via prefix continuation and derives a closed-form KL divergence for single continuations. From this analysis the authors propose Adaptive Reference Guidance (ARG), a learned prefix selector that constructs correct trajectories within a fixed generation budget and, when applied to GRPO with Qwen3-4B and Qwen3-8B across five mathematical reasoning benchmarks, achieves the highest aggregate pass@12 among evaluated methods with competitive sampled accuracy.
KEY POINTS
- The arXiv preprint (2610.11128v1) analyzes how much reference guidance to provide when training reasoning models via prefix continuation and derives a closed-form KL divergence for single continuations.
- From this analysis the authors propose Adaptive Reference Guidance (ARG), a learned prefix selector that constructs correct trajectories within a fixed generation budget and, when applied to GRPO with Qwen3-4B and Qwen3-8B across five mathematical reasoning benchmarks, achieves the highest aggregate pass@12 among evaluated methods with competitive sampled accuracy.
- ARG offers a practical, analysis-backed way to adaptively choose reference prefix length—improving verified trajectory generation efficiency and pass@12 performance without estimating success probabilities.
WHY IT MATTERS
ARG offers a practical, analysis-backed way to adaptively choose reference prefix length—improving verified trajectory generation efficiency and pass@12 performance without estimating success probabilities.