Cohere introduces hardware-aware Dynamic Speculative Decoding (DSD) to adapt draft-token count to GPU constraints
Cohere presents Dynamic Speculative Decoding (DSD), an extension of speculative decoding that selects the number of draft tokens K based on hardware regime (memory-bandwidth vs compute-bound) and model type (dense vs MoE). DSD uses offline profiling of Acceptance Length (AL) and Inter-Token Latency (ITL) to choose an optimal K per condition, aiming to preserve SD speedups in production workloads and large-scale RL rollouts.