RESEARCH · RESEARCH · #644
TS-DFM: energy-guided distillation cuts discrete flow matching to 8 steps with better perplexity
Trajectory-Shaped Discrete Flow Matching (TS-DFM) replaces the blind stochastic mid-step jumps used to build training trajectories with a lightweight 'energy compass' that selects more coherent continuations during distillation. On a 170M-parameter language model the TS-DFM student at 8 generation steps achieves 32% lower perplexity than its 1,024-step teacher while running 128× faster, with gains consistent across data sources and multiple evaluators; the shaping is applied only during training so inference cost is unchanged.
KEY POINTS
- Trajectory-Shaped Discrete Flow Matching (TS-DFM) replaces the blind stochastic mid-step jumps used to build training trajectories with a lightweight 'energy compass' that selects more coherent continuations during distillation.
- On a 170M-parameter language model the TS-DFM student at 8 generation steps achieves 32% lower perplexity than its 1,024-step teacher while running 128× faster, with gains consistent across data sources and multiple evaluators; the shaping is applied only during training so inference cost is unchanged.
- By shaping distillation trajectories with an inexpensive evaluator, TS-DFM enables much faster few-step discrete-generation with higher quality, addressing a key bottleneck for parallel non-autoregressive language generation.
WHY IT MATTERS
By shaping distillation trajectories with an inexpensive evaluator, TS-DFM enables much faster few-step discrete-generation with higher quality, addressing a key bottleneck for parallel non-autoregressive language generation.