RESEARCH · RESEARCH · #1882
NVIDIA and Dao AI Lab publish ReplaySSM to speed SSM-based hybrid-model decoding
NVIDIA and the Dao AI Lab (Princeton) released ReplaySSM (paper on arXiv and code on GitHub), a technique that caches recent SSM inputs instead of full state to halve state memory traffic and speed SSM decoding kernels by ~1.46–1.80× (batch 256 on Blackwell Ultra B300). ReplaySSM also simplifies speculative-decoding rollback (1.85–2.10× higher speculative throughput on 120B-class models) and enables mixed-precision state (three of four tested models stay within ~2 percentage points of FP32 on GPQA‑Diamond); it's integrated into TensorRT‑LLM, vLLM, and FlashInfer and runs in production in GLM‑5.3‑Flash and Kimi‑K3.
KEY POINTS
- NVIDIA and the Dao AI Lab (Princeton) released ReplaySSM (paper on arXiv and code on GitHub), a technique that caches recent SSM inputs instead of full state to halve state memory traffic and speed SSM decoding kernels by ~1.46–1.80× (batch 256 on Blackwell Ultra B300).
- ReplaySSM also simplifies speculative-decoding rollback (1.85–2.10× higher speculative throughput on 120B-class models) and enables mixed-precision state (three of four tested models stay within ~2 percentage points of FP32 on GPQA‑Diamond); it's integrated into TensorRT‑LLM, vLLM, and FlashInfer and runs in production in GLM‑5.3‑Flash and Kimi‑K3.
- Makes SSM-heavy hybrid models faster and more memory-efficient at serving scale, restoring speculative decoding and practical quantization for production inference.
WHY IT MATTERS
Makes SSM-heavy hybrid models faster and more memory-efficient at serving scale, restoring speculative decoding and practical quantization for production inference.