RESEARCH · RESEARCH · #645
DACA-GRPO: denoising-aware credit assignment improves GRPO for diffusion LLMs
The paper introduces DACA-GRPO, a lightweight plug-and-play enhancement for GRPO-style reinforcement learning on diffusion language models that addresses missing temporal credit assignment and mean-field likelihood bias. It adds Denoising Progress Scores (per-token importance weights from intermediate predictions) and Stratified Masking Likelihood (token strata to reduce mean-field bias), and reports consistent gains across seven benchmarks—up to 5.6 percentage points on math reasoning, 7.4 pp on code generation, 36.3 pp on constraint satisfaction, and 5.9 pp on JSON schema adherence.
KEY POINTS
- The paper introduces DACA-GRPO, a lightweight plug-and-play enhancement for GRPO-style reinforcement learning on diffusion language models that addresses missing temporal credit assignment and mean-field likelihood bias.
- It adds Denoising Progress Scores (per-token importance weights from intermediate predictions) and Stratified Masking Likelihood (token strata to reduce mean-field bias), and reports consistent gains across seven benchmarks—up to 5.6 percentage points on math reasoning, 7.4 pp on code generation, 36.3 pp on constraint satisfaction, and 5.9 pp on JSON schema adherence.
- By adding per-step credit assignment and reducing mean-field likelihood bias, DACA-GRPO materially improves RL optimization for diffusion LLMs, yielding substantial gains on reasoning and constrained-generation tasks.
WHY IT MATTERS
By adding per-step credit assignment and reducing mean-field likelihood bias, DACA-GRPO materially improves RL optimization for diffusion LLMs, yielding substantial gains on reasoning and constrained-generation tasks.