NEWS · RESEARCH · #368
Calibrate, Then Route: learned request routing improves goodput for disaggregated LLM serving (arXiv:2609.16206v1)
The paper "Calibrate, Then Route" (arXiv:2609.16206v1) studies a learned router for disaggregated LLM serving that estimates per-instance completion time using prompt/output lengths, KV cache pressure, and SLO class. Implemented in a discrete-event simulator and validated on eight NVIDIA A40 GPUs running vLLM with NIXL for KV transfers, the calibrated router attained the highest mean goodput (0.864) across three mixed, bursty traces vs. 0.835–0.847 for round robin, least-loaded, and a length heuristic, with lower variance; calibration accounted for most of the tail-latency benefit and could reduce required decode GPUs (matching round robin goodput with six vs seven GPUs).
KEY POINTS
- The paper "Calibrate, Then Route" (arXiv:2609.16206v1) studies a learned router for disaggregated LLM serving that estimates per-instance completion time using prompt/output lengths, KV cache pressure, and SLO class.
- Implemented in a discrete-event simulator and validated on eight NVIDIA A40 GPUs running vLLM with NIXL for KV transfers, the calibrated router attained the highest mean goodput (0.864) across three mixed, bursty traces vs.
- 0.835–0.847 for round robin, least-loaded, and a length heuristic, with lower variance; calibration accounted for most of the tail-latency benefit and could reduce required decode GPUs (matching round robin goodput with six vs seven GPUs).
WHY IT MATTERS
This matters because calibrated, learned routing measurably improves LLM serving efficiency and latency tails, potentially reducing GPU requirements for decode pools in production deployments.