RESEARCH · RESEARCH · #1630
SPECTRUM: Proximal Spectral Modulation improves diversity retention in looped self-distillation for code generation
This arXiv paper introduces Looped Self-Distillation, a framework where a model repeatedly generates and learns from its own outputs under a fixed information budget, and presents SPECTRUM — a method that re-estimates loss-sensitive key/value geometry from a fixed reference and applies full-rank proximal spectral modulation. In experiments on MBPP, SPECTRUM preserved 89.9% of the initial model's 64-sample correct-AST richness after five rounds (vs. 66.4% for vanilla self-distillation), and the diversity improvement transferred to HumanEval+ and APPS Intro without additional recalibration.
KEY POINTS
- This arXiv paper introduces Looped Self-Distillation, a framework where a model repeatedly generates and learns from its own outputs under a fixed information budget, and presents SPECTRUM — a method that re-estimates loss-sensitive key/value geometry from a fixed reference and applies full-rank proximal spectral modulation.
- In experiments on MBPP, SPECTRUM preserved 89.9% of the initial model's 64-sample correct-AST richness after five rounds (vs.
- 66.4% for vanilla self-distillation), and the diversity improvement transferred to HumanEval+ and APPS Intro without additional recalibration.
WHY IT MATTERS
Shows that generation-time spectral intervention can preserve a model's repertoire of correct solutions during recursive self-improvement, introducing 'correct-solution retention' as a complementary training objective for iterative self-distillation.