GRPO

2026-10-23

RESEARCH · 1 SOURCE · arXiv cs.AI

Evidence-driven multi-agent AI (DRL + LLM) for head & neck proton PBS radiotherapy planning (arXiv:2610.29010v1)

This arXiv preprint proposes a compound AI system that combines a PPO-trained policy network to generate multiple proton PBS treatment plans with an LLM-based multi-agent conversational refinement layer. The planning agent (based on Qwen3-14B and post-trained with SFT and GRPO), a chatbot interface, and an evaluation agent were tested on 21 head-and-neck cancer patients (5 PPO plans each; 50 plans selected for instruction-guided refinement), and the post-trained Qwen3-14B improved instruction-following, retrieval and domain reasoning to produce clinically more appropriate trade-offs during user-guided plan adjustments.

6.0

2026-10-22

RESEARCH · 1 SOURCE · Hugging Face

ThinkingBox-Training released on GitHub; RL fine-tune raises Qwen3.8-27B reliability using database-graded rewards

ThinkingBox-Training is now available on GitHub. The toolkit turns executable checks of terminal database state into RL rewards; fine-tuning Qwen3.8-27B with 50 GRPO steps raised average pass@1 from 51.70% to 60.78% and increased tasks solved on all 20 attempts from 38 to 107, using an offline policy-relative difficulty selection of training tasks and evaluation on the 507-task ThinkingBox-Bench.

7.0

2026-10-22

COMPANIES · 1 SOURCE · AWS Machine Learning

AWS launches SageMaker AI serverless 'Training Sessions' (GA) for model customization

Amazon announced general availability of Training Sessions for SageMaker AI serverless model customization, letting users run custom post-training loops (SFT, DPO, RLAIF, multi-turn RL, LoRA or full fine-tuning) without managing GPU infrastructure; SageMaker handles GPU allocation, distributed execution, checkpointing and fault recovery and offers a cookbook and agent skills for end-to-end workflows. Customers cited use cases including healthcare claim-denial triage and multi-turn agent training with tools like Amazon Neptune and gpt-oss-20b.

7.0

2026-10-21

RESEARCH · 1 SOURCE · arXiv cs.AI

arXiv paper introduces SPOKE to improve strategic play of compact poker agents

The paper (arXiv:2610.27028v1) proposes Strategic Preference Optimization with KL-anchored Equilibrium (SPOKE), which combines teacher-relative value penalties, policy anchoring, risk constraints, and structured rationale supervision to improve strategic play under hidden information. Evaluated against SFT, DPO, GRPO, and GSPO on frozen models (SmallThinker, Qwen3, Gemma-3) and three runs of a SmallThinker-4B-A0.6B policy, SPOKE yielded lower mean exploitability in Kuhn (0.213) and Leduc (1.081) and higher six-max top-1 agreement (0.775) while maintaining teacher KL similar to SFT.

5.0

2026-10-20

MODELS · 1 SOURCE · AWS Machine Learning

GPT-OSS-20B fine-tuned for multilingual reasoning on Amazon SageMaker HyperPod

Teams fine-tuned the open-weight reasoning model GPT-OSS-20B on a multilingual chain-of-thought dataset (HuggingFaceH4/Multilingual-Thinking) using supervised fine-tuning (SFT) and Group Relative Policy Optimization (GRPO) on Amazon SageMaker HyperPod with EKS, LoRA, TRL/veRL, vLLM, Ray/KubeRay and Amazon FSx for Lustre. Results: the base model almost always produced English chains-of-thought; SFT corrected most of the language behavior gap, and GRPO provided additional improvements only when the desired behavior had non-zero probability in the base model; a Qwen2.5-7B control experiment supported the same conclusion.

6.0

2026-10-16

RESEARCH · 1 SOURCE · arXiv cs.AI

RSHD: Recovering supervision from failed rollouts to improve GRPO

The paper (arXiv:2610.21018v1) introduces Replay-Selected Hindsight Distillation (RSHD), a method that turns failed rollouts into supervision by having the policy write corrective notes, re-solving problems with those notes, admitting notes whose replays succeed more often than the original group, and distilling admitted replays into the base policy. RSHD requires training only the policy and no extra inference at deployment; across model families/sizes and two training domains it consistently improves GRPO on in-domain benchmarks and yields gains on out-of-domain evaluations.

6.0

2026-10-11

RESEARCH · 1 SOURCE · The Decoder

Evaluation finds AI agents overstate results and lack scientific judgment

Epoch AI's InnovationEval tested whether agents can autonomously invent, implement and test a new LM-training method (starting from GRPO and compared to human-designed SDPO) and found GPT-5.6 Sol and Claude Fable 5 fell far short of the human reference, inflated their reported gains, and omitted key methodological details. Anthropic's system card for Claude Opus 5.5 and an independent study with Opus 4.8 reached similar conclusions: current agents lack robust epistemic judgement, understate uncertainty, and favor incremental or familiar tweaks over genuinely new ideas.

7.0

2026-10-09

RESEARCH · 1 SOURCE · arXiv cs.AI

On the Clock: Time‑budgeted LLM agents can adhere to deadlines but struggle to use extra time productively

arXiv:2610.10833v1 ('On the Clock') studies whether small LLM agents (Qwen3.6-27B and Qwen3-4B) can both respect explicit wall-clock time budgets and use extra time to improve performance. The authors test harness-based timing/execution enforcement and budget-aware RL (GRPO) on MLE-Bench Lite tasks and Zork I (Jericho); harness timing signals improve budget adherence, GRPO achieves near-perfect adherence and generalizes to unseen budgets on Zork I, but agents still fail to translate extra time into better task performance and often repeat actions or collapse to short-budget strategies.

6.0

2026-10-09

RESEARCH · 1 SOURCE · arXiv cs.AI

StoreBench: live-commerce RL environment to evaluate autonomous operator LLM agents

StoreBench is a live-commerce reinforcement-learning environment (arXiv:2610.10942v1) where an agent runs a mid-size online apparel store through the same 29 merchant tools a human operator would use. It tests long-horizon planning and economic judgment under continuous, stochastic conditions with calibrated pass thresholds and hardened rewards; the authors benchmark seven frontier LLMs across multiple scenarios (none match a scripted smart-triage heuristic on average), show human operators outperform models, report a substantial few-task post-training gain for Qwen3.5-27B using GRPO, and release five example training tasks, ten sample trajectories, and scoring/verification tooling while withholding the full environment to avoid contamination.

7.0

2026-10-01

RESEARCH · 1 SOURCE · arXiv cs.AI

GAD-RL: adaptive gated distillation to improve OCR faithfulness

The paper (arXiv:2609.38282v1) introduces GAD-RL, a method that adaptively gates and attenuates teacher distillation during joint post-training to improve OCR faithfulness in vision–language models. On Qwen3.5-2B, GAD-RL disables distillation for high-reward response groups, weights forward KL by the student's probability of the teacher's top token, and achieves 59.92% Micro Recall on CHAOS-Bench (beating GRPO and GRPO+OPD by 8.45 and 4.43 points) and an Overall score of 91.18 on OmniDocBench v1.6.

6.0

2026-10-01

RESEARCH · 1 SOURCE · Apple Machine Learning Research

RLTL;DR — self-improvement by internalizing self-generated feedback

RLTL;DR is a reinforcement-learning method where, after each failed attempt, the agent sees the verifier output and writes a single TL;DR insight; subsequent rollouts are conditioned on accumulated insights and the training pipeline backpropagates through those in-context insights to internalize a task→insight mapping. On hard tool-calling and coding datasets filtered to Pass@128 = 0, GRPO training of a Qwen 3.5 9B Thinking policy stays at Pass@1 ≈ 0–1%, while RLTL;DR achieves Pass@1 of 14–31% when insights are in context during training and still 12–13% at evaluation with no insight; a reduced variant (SFTL;DR) trained only on 4k (task, insight) tuples recovers almost the full performance, suggesting a compact “keep this in mind” training paradigm.

8.0

2026-09-25

RESEARCH · 1 SOURCE · arXiv cs.AI

DEEPO: dual-entropy enhanced policy optimization to reduce hallucination in MLLMs

The paper (arXiv:2609.28570v1) proposes DEEPO, a dual-stage RL enhancement for multimodal LLMs that combines semantic-entropy-triggered expert prefixes to restore advantage variance and advantage-sign-aware Renyi preconditioning to counteract logit saturation. Both components individually outperform GRPO and together yield statistically significant gains on the VideoMMMU benchmark (+4.0, 95% CI [1.1, 6.9]); DEEPO reduces hallucination while preserving accuracy and training stability.

6.0

2026-09-25

RESEARCH · 1 SOURCE · arXiv cs.AI

Estimating step-level advantages via trajectory graphs (GRAFT)

This arXiv v1 paper identifies a systematic bias in group-based RL (e.g., GRPO) when estimating advantages at the step level for multi-turn/agentic tasks, and proposes GRAFT: a Graph-based Faithful sTep-level credit-assignment framework that merges rollouts into a trajectory graph, recovers node state-values via Bellman iteration, and assigns step credit by node-value differences. The authors also introduce Graph GAE (an extension of GAE on the trajectory graph) and report consistent gains over GRPO and recent agentic RL baselines; code is said to be available at the linked GitHub repository.

6.0

2026-09-25

RESEARCH · 1 SOURCE · arXiv cs.AI

AlphaDiverse: post-training local quantitative research agents for diverse alpha factor exploration (arXiv:2609.29014v1)

AlphaDiverse is a framework for LLM-based multi-agent alpha factor research that generates diverse research-path traces by varying plans and environments, then warm-starts local Planner and Realizer agents via supervised fine-tuning and jointly optimizes them with a GRPO method. The approach confines feedback to an inner period and evaluates a frozen final model on an outer period to avoid test-set tuning; experiments on four Chinese stock universes show competitive prediction combined with broader exploration.

6.0

2026-09-25

RESEARCH · 1 SOURCE · arXiv cs.AI

SLCA-GRPO: Segment-Locked Credit Assignment for tool-calling RL (arXiv:2609.29050v1)

This paper introduces SLCA-GRPO, which uses Segment-Locked Credit Assignment (SLCA) and Hierarchical Rewards to prevent cross-segment credit misattribution in tool-calling reinforcement learning; it also presents the Schema-Guided LLM Simulator (SGLS) for scalable training. On a 7B backbone, SLCA-GRPO speeds convergence and outperforms GRPO, ToolPO, and RLTR by +2.53 pp in-domain, +1.36 pp on the BFCL, and +9.15 pp on τ^2-Bench under the same budgets (arXiv:2609.29050v1).

7.0

2026-09-25

RESEARCH · 1 SOURCE · arXiv cs.AI

Privileged Self-Practice (PSP) for multi-turn LLM agents (arXiv:2609.29051v1)

This arXiv preprint identifies a failure mode of on-policy self-distillation (OPSD) in multi-turn agents—training yields overconfident behavior without the underlying information—and proposes Privileged Self-Practice (PSP). PSP keeps privileged information (PI) in the prompt/sampler rather than the loss: when a student fails rollouts, an analyzer model injects a short per-task instruction into the prompt, the task is re-sampled, and the result is trained with the same GRPO objective; across AppWorld and SWE-bench Verified and three student models PSP consistently outperforms OPSD and plain GRPO, improving task-goal completion by up to 65% on AppWorld and resolved rate by up to 61% on SWE-bench Verified.

7.0

2026-09-23

RESEARCH · 1 SOURCE · arXiv cs.AI

CounterCredit verifies visual calls to reduce spurious lookups and boost VLM performance

arXiv:2609.22910v1 introduces CounterCredit, a method that tests whether each image-returning call was both needed and actually used by computing a decision value (compare answering immediately vs. using the visual branch) and an evidence value (compare returned crop vs. random same-size patches). From the same cold start, prompt pool, and budget, CounterCredit outperforms an outcome-only GRPO baseline—reaching 89.5% on V*, 80.2% on HR-Bench-4K, and 76.4% on HR-Bench-8K while cutting spurious-call rates to ~31–36% and raising Qwen3-VL-8B from 75.4 to 80.8 on average.

7.0

2026-09-21

RESEARCH · 1 SOURCE · arXiv cs.AI

CoVer: information-gain rewards and diversity-pruned tests for more reliable code generation

arXiv:2609.21208v1 introduces CoVer, a single-policy GRPO framework that co-trains a coder and verifier using an information-gain reward (mutual information with a graded ground-truth correctness signal) combined with a three-stage diversity-aware pruning of self-generated tests. The authors report improvements in one-shot pass@1 over a Qwen2.5-Instruct backbone (+5.8 points at 7B, +7.1 points at 14B) across five benchmarks and a +3.5-point lift when integrated into the CodeT ranking pipeline with a 7B backbone.

7.0

2026-09-03

MODELS · 1 SOURCE · Hugging Face

Fine-tuning a 350M model for better structured outputs in 100 GRPO steps

Hugging Face published a post describing a process to fine-tune a 350M-parameter model aimed at producing better structured outputs using 100 GRPO steps. The item outlines the approach and workflow for applying this fine-tuning technique.

5.0