RESEARCH · RESEARCH · #1334
GoldiMask: context selection and target weighting for fine-tuning diffusion language models
arXiv:2609.38385v1 introduces GoldiMask, a supervised fine-tuning procedure for discrete diffusion language models that selects which tokens to reveal as context via an approximate submodular objective and weights remaining prediction targets by their benefit and learnability. Across three backbones and three datasets the paper reports higher average accuracy in most settings (including reasoning and code generation) and reduced decoding iterations on GSM8K and MATH-500 under confidence-threshold parallel decoding; ablations show both context selection and target weighting contribute to the gains.
KEY POINTS
- arXiv:2609.38385v1 introduces GoldiMask, a supervised fine-tuning procedure for discrete diffusion language models that selects which tokens to reveal as context via an approximate submodular objective and weights remaining prediction targets by their benefit and learnability.
- Across three backbones and three datasets the paper reports higher average accuracy in most settings (including reasoning and code generation) and reduced decoding iterations on GSM8K and MATH-500 under confidence-threshold parallel decoding; ablations show both context selection and target weighting contribute to the gains.
- Optimizing which tokens are revealed and how targets are weighted can improve accuracy and decoding efficiency when fine-tuning diffusion language models, affecting downstream reasoning and code-generation performance.
WHY IT MATTERS
Optimizing which tokens are revealed and how targets are weighted can improve accuracy and decoding efficiency when fine-tuning diffusion language models, affecting downstream reasoning and code-generation performance.