Group Relative Policy Optimization (GRPO)

2026-10-09

RESEARCH · 1 SOURCE · arXiv cs.AI

Adaptive Reference Guidance (ARG) for prefix continuation in reasoning RL (arXiv:2610.11128v1)

The arXiv preprint (2610.11128v1) analyzes how much reference guidance to provide when training reasoning models via prefix continuation and derives a closed-form KL divergence for single continuations. From this analysis the authors propose Adaptive Reference Guidance (ARG), a learned prefix selector that constructs correct trajectories within a fixed generation budget and, when applied to GRPO with Qwen3-4B and Qwen3-8B across five mathematical reasoning benchmarks, achieves the highest aggregate pass@12 among evaluated methods with competitive sampled accuracy.

6.0

2026-09-25

COMPANIES · 1 SOURCE · AWS Machine Learning

Amazon describes EKS + EFA + DeepEP architecture to scale MoE reinforcement learning with up to 40% higher throughput

An Amazon post details an architecture that combines Amazon EKS, Elastic Fabric Adapter (EFA), and DeepEP to address the heterogeneous compute, high-bandwidth communication, and orchestration challenges of post-training MoE reinforcement learning (including RLHF and GRPO). The write-up explains balancing rollout generation and tightly coupled policy training, optimizing Expert Parallelism (EP) communication over EFA, and reports up to a 40% throughput improvement using DeepEP.

6.0

2026-09-25

COMPANIES · 1 SOURCE · AWS Machine Learning

Run SkyRL GRPO post-training of Qwen3‑VL‑8B on Amazon SageMaker HyperPod

Amazon demonstrates running the open-source SkyRL framework on SageMaker HyperPod (EKS) to post-train a Qwen3‑VL‑8B vision‑language model with Group Relative Policy Optimization (GRPO). Using a HyperPod Ray cluster with shared FSx storage, the workflow improved maze solve rate from 43.75% to over 95% on a fixed 64‑maze evaluation set and uses colocated vLLM inference and FSDP-sharded policy training with LoRA adapters.

6.0

2026-09-25

RESEARCH · 1 SOURCE · arXiv cs.AI

Paper shows RLVR (with GRPO) works on Qwen3.5-0.8B without distillation

The arXiv preprint applies Reinforcement Learning with Verifiable Rewards (RLVR) using Group Relative Policy Optimization (GRPO) and an interleaved Wikipedia search tool to train Qwen3.5-0.8B on MuSiQue and a seven-benchmark QA suite. The best run reaches 0.352 average exact-match (vs 0.092 untrained), a 3.8× improvement with no distillation, and the authors find reward shape matters—sparse exact-match rewards perform worst for this model scale.

6.0

2026-09-24

RESEARCH · 1 SOURCE · arXiv cs.AI

RLDS: decomposing trajectory rewards by subtask improves RL for language-model agents (arXiv:2609.27035v1)

The paper introduces Reinforcement Learning with Decomposed Subtasks (RLDS) and Subtask-Decomposed Advantage Estimation (SDAE), which split trajectory reward into per-subtask shares before policy updates instead of collapsing outcomes to a single scalar as in Group Relative Policy Optimization (GRPO). Evaluated on four benchmarks, RLDS produced substantial gains where subtask heterogeneity is high (ScienceWorld +11.5 points, paired-bootstrap 95% CI [+9.8, +13.3]; FrozenLake +9.8 points, CI [+7.0, +12.8]), showed little effect where diagnostics predicted little recovery (HotpotQA, DeepResearch), and was more compute-efficient on ScienceWorld (-10.9% wall-clock per step).

7.0

2026-09-16

RESEARCH · 1 SOURCE · Apple Machine Learning Research

DACA-GRPO: denoising-aware credit assignment improves GRPO for diffusion LLMs

The paper introduces DACA-GRPO, a lightweight plug-and-play enhancement for GRPO-style reinforcement learning on diffusion language models that addresses missing temporal credit assignment and mean-field likelihood bias. It adds Denoising Progress Scores (per-token importance weights from intermediate predictions) and Stratified Masking Likelihood (token strata to reduce mean-field bias), and reports consistent gains across seven benchmarks—up to 5.6 percentage points on math reasoning, 7.4 pp on code generation, 36.3 pp on constraint satisfaction, and 5.9 pp on JSON schema adherence.

7.0