Tech Meridian ← ENTITY INDEX
RU

TOPIC · ENTITY #1361

distillation

Related event timeline, sources and context from the news index.

EVENT TIMELINE

3

RESEARCH · 1 SOURCE · Apple Machine Learning Research

TS-DFM: energy-guided distillation cuts discrete flow matching to 8 steps with better perplexity

Trajectory-Shaped Discrete Flow Matching (TS-DFM) replaces the blind stochastic mid-step jumps used to build training trajectories with a lightweight 'energy compass' that selects more coherent continuations during distillation. On a 170M-parameter language model the TS-DFM student at 8 generation steps achieves 32% lower perplexity than its 1,024-step teacher while running 128× faster, with gains consistent across data sources and multiple evaluators; the shaping is applied only during training so inference cost is unchanged.

7.0

RESEARCH · 1 SOURCE · Apple Machine Learning Research

PROOF-Gen: From optimized training data to improved distillation for tool-calling

The paper presents PROOF-Gen, a method that focuses on generating optimized training data to improve supervised fine-tuning of student models distilled from teacher-generated trajectories, aiming to overcome limitations of the common generate-and-filter pipeline that leaves hard failure cases unaddressed. The authors report that on τ 2-bench, 57% of teacher trials fail and roughly two-thirds of those failures are near-misses (most tool calls correct but subsequently undone), motivating the need to extract signal from failures rather than discarding them.

7.0

RESEARCH · 1 SOURCE · Google Research

Google Research proposes 'early readouts' and a 'feature sieve' to mitigate spurious features and simplicity bias

Google Research authors Rishabh Tiwari and Pradeep Shenoy describe two interventions—using predictions from early intermediate layers ('early readouts') and inducing 'feature forgetting' via a 'feature sieve'—to detect and reduce reliance on spurious features and the simplicity bias in deep networks. They apply the early-readout signal to reweight teacher contributions during distillation and use a feature-sieving intervention to encourage models to find more predictive features, reporting improved worst-group accuracy across benchmarks (Waterbirds, CelebA, CivilComments, MNLI) and better generalization to unseen domains compared with prior methods.

7.0