Tech Meridian ← LIVE FEED
PROMY MERIDIAN RU

NEWS · CODING · #2208

vLLM adds push-based NIXL connector to speed KV transfer for disaggregated inference

vLLM introduced a push-based NIXL connector that moves KV-transfer coordination off the decode critical path for prefill/decode disaggregated inference. In benchmarks on EC2 P5en instances with EFA, push mode reportedly reduced time-to-first-token by up to 3× and improved time-per-output-token by up to 30% for Qwen3-32B, with results varying by workload and configuration.

KEY POINTS

  1. vLLM introduced a push-based NIXL connector that moves KV-transfer coordination off the decode critical path for prefill/decode disaggregated inference.
  2. In benchmarks on EC2 P5en instances with EFA, push mode reportedly reduced time-to-first-token by up to 3× and improved time-per-output-token by up to 30% for Qwen3-32B, with results varying by workload and configuration.
  3. Efficient KV transfer is a bottleneck for disaggregated prefill/decode inference, so moving transfer setup off the critical path can materially reduce latency and increase throughput for serving large models.

WHY IT MATTERS

Efficient KV transfer is a bottleneck for disaggregated prefill/decode inference, so moving transfer setup off the critical path can materially reduce latency and increase throughput for serving large models.

SOURCES & TIMELINE

1
01
AWS MACHINE LEARNING FIRST-PARTY
Push-based KV transfer to improve TTFT and TPOT for disaggregated inference in vLLM

vLLM is an open-source inference and serving engine for large language models (LLMs). LLM inference has two distinct phases: prefill, which processes the input prompt and builds the key-value (KV) cache, and decode, which uses that cache to generate output one token at a time. These phases have different compute, memory, and scaling characteristics. Prefill-decode (P-D) disaggregation runs them on separate engine ins…

↗