Tech Meridian ← ENTITY INDEX
PROMY MERIDIAN RU

TOPIC · ENTITY #20286

prefill-decode disaggregation

Related event timeline, sources and context from the news index.

EVENT TIMELINE

2

CODING · 1 SOURCE · AWS Machine Learning

vLLM adds push-based NIXL connector to speed KV transfer for disaggregated inference

vLLM introduced a push-based NIXL connector that moves KV-transfer coordination off the decode critical path for prefill/decode disaggregated inference. In benchmarks on EC2 P5en instances with EFA, push mode reportedly reduced time-to-first-token by up to 3× and improved time-per-output-token by up to 30% for Qwen3-32B, with results varying by workload and configuration.

6.0

CODING · 1 SOURCE · AWS Machine Learning

vLLM adds bi-directional KV transfer to reduce TTFT for multi-turn agent workloads

vLLM implemented bi-directional key-value (KV) cache transfer so decode nodes can return cached KV blocks to prefill nodes across turns, avoiding repeated recomputation of earlier responses and context. In 1P1D Amazon EC2 P5en+EFA benchmarks the change reduced time-to-first-token (TTFT) by up to 1.6x for Qwen3-32B and up to 3.3x for zai-org/GLM-5-FP8; results vary by workload and config.

7.0