FUNDING · MODELS · #3374
Attention to Mamba: two-stage recipe distills Pythia-1B Transformer into Mamba SSM
Authors propose a principled two-stage cross-architecture distillation: first distill a Transformer into a linearized-Attention model (using a kernel-trick adaptation), then distill that into a Mamba-style State Space Model without Attention. The distilled Mamba nearly matches the Pythia-1B teacher (perplexity 14.01 vs 13.86) across evaluation and includes ablations, scaling analyses, and a distillation scaling law for compute-optimal allocation.
KEY POINTS
- Authors propose a principled two-stage cross-architecture distillation: first distill a Transformer into a linearized-Attention model (using a kernel-trick adaptation), then distill that into a Mamba-style State Space Model without Attention.
- The distilled Mamba nearly matches the Pythia-1B teacher (perplexity 14.01 vs 13.86) across evaluation and includes ablations, scaling analyses, and a distillation scaling law for compute-optimal allocation.
- This provides a practical recipe and initialization to transfer widely available pretrained Transformers into Mamba SSMs with near-teacher performance and offers compute-optimal distillation guidance, lowering the barrier to adopt SSM architectures.
WHY IT MATTERS
This provides a practical recipe and initialization to transfer widely available pretrained Transformers into Mamba SSMs with near-teacher performance and offers compute-optimal distillation guidance, lowering the barrier to adopt SSM architectures.