NEWS · RESEARCH · #420
Cohere introduces hardware-aware Dynamic Speculative Decoding (DSD) to adapt draft-token count to GPU constraints
Cohere presents Dynamic Speculative Decoding (DSD), an extension of speculative decoding that selects the number of draft tokens K based on hardware regime (memory-bandwidth vs compute-bound) and model type (dense vs MoE). DSD uses offline profiling of Acceptance Length (AL) and Inter-Token Latency (ITL) to choose an optimal K per condition, aiming to preserve SD speedups in production workloads and large-scale RL rollouts.
KEY POINTS
- Cohere presents Dynamic Speculative Decoding (DSD), an extension of speculative decoding that selects the number of draft tokens K based on hardware regime (memory-bandwidth vs compute-bound) and model type (dense vs MoE).
- DSD uses offline profiling of Acceptance Length (AL) and Inter-Token Latency (ITL) to choose an optimal K per condition, aiming to preserve SD speedups in production workloads and large-scale RL rollouts.
- DSD matters because it makes speculative decoding practical across varying batch sizes and MoE architectures, improving GPU utilization and maintaining inference speedups in production and RL rollout settings.
WHY IT MATTERS
DSD matters because it makes speculative decoding practical across varying batch sizes and MoE architectures, improving GPU utilization and maintaining inference speedups in production and RL rollout settings.