Tech Meridian ← ENTITY INDEX
PROMY MERIDIAN RU

TOPIC · ENTITY #31782

distillation scaling law

Related event timeline, sources and context from the news index.

EVENT TIMELINE

1

MODELS · 1 SOURCE · Apple Machine Learning Research

Attention to Mamba: two-stage recipe distills Pythia-1B Transformer into Mamba SSM

Authors propose a principled two-stage cross-architecture distillation: first distill a Transformer into a linearized-Attention model (using a kernel-trick adaptation), then distill that into a Mamba-style State Space Model without Attention. The distilled Mamba nearly matches the Pythia-1B teacher (perplexity 14.01 vs 13.86) across evaluation and includes ablations, scaling analyses, and a distillation scaling law for compute-optimal allocation.

7.0