NEWS · CODING · #2207
vLLM adds bi-directional KV transfer to reduce TTFT for multi-turn agent workloads
vLLM implemented bi-directional key-value (KV) cache transfer so decode nodes can return cached KV blocks to prefill nodes across turns, avoiding repeated recomputation of earlier responses and context. In 1P1D Amazon EC2 P5en+EFA benchmarks the change reduced time-to-first-token (TTFT) by up to 1.6x for Qwen3-32B and up to 3.3x for zai-org/GLM-5-FP8; results vary by workload and config.
KEY POINTS
- vLLM implemented bi-directional key-value (KV) cache transfer so decode nodes can return cached KV blocks to prefill nodes across turns, avoiding repeated recomputation of earlier responses and context.
- In 1P1D Amazon EC2 P5en+EFA benchmarks the change reduced time-to-first-token (TTFT) by up to 1.6x for Qwen3-32B and up to 3.3x for zai-org/GLM-5-FP8; results vary by workload and config.
- Reducing redundant KV recomputation across turns lowers accelerator usage and keeps TTFT responsive for growing multi-turn, agentic conversations.
WHY IT MATTERS
Reducing redundant KV recomputation across turns lowers accelerator usage and keeps TTFT responsive for growing multi-turn, agentic conversations.