Tech Meridian ← LIVE FEED
PROMY MERIDIAN RU

FUNDING · MODELS · #3374

Attention to Mamba: two-stage recipe distills Pythia-1B Transformer into Mamba SSM

Authors propose a principled two-stage cross-architecture distillation: first distill a Transformer into a linearized-Attention model (using a kernel-trick adaptation), then distill that into a Mamba-style State Space Model without Attention. The distilled Mamba nearly matches the Pythia-1B teacher (perplexity 14.01 vs 13.86) across evaluation and includes ablations, scaling analyses, and a distillation scaling law for compute-optimal allocation.

KEY POINTS

  1. Authors propose a principled two-stage cross-architecture distillation: first distill a Transformer into a linearized-Attention model (using a kernel-trick adaptation), then distill that into a Mamba-style State Space Model without Attention.
  2. The distilled Mamba nearly matches the Pythia-1B teacher (perplexity 14.01 vs 13.86) across evaluation and includes ablations, scaling analyses, and a distillation scaling law for compute-optimal allocation.
  3. This provides a practical recipe and initialization to transfer widely available pretrained Transformers into Mamba SSMs with near-teacher performance and offers compute-optimal distillation guidance, lowering the barrier to adopt SSM architectures.

WHY IT MATTERS

This provides a practical recipe and initialization to transfer widely available pretrained Transformers into Mamba SSMs with near-teacher performance and offers compute-optimal distillation guidance, lowering the barrier to adopt SSM architectures.

SOURCES & TIMELINE

1