Tech Meridian
← ENTITY INDEX
TOPIC · ENTITY #477
Related event timeline, sources and context from the news index.
vLLM implemented bi-directional key-value (KV) cache transfer so decode nodes can return cached KV blocks to prefill nodes across turns, avoiding repeated recomputation of earlier responses and context. In 1P1D Amazon EC2 P5en+EFA benchmarks the change reduced time-to-first-token (TTFT) by up to 1.6x for Qwen3-32B and up to 3.3x for zai-org/GLM-5-FP8; results vary by workload and config.
This arXiv preprint introduces OrchSLM, a routing framework that unifies existing non-interactive orchestration methods for small language models (SLMs) and exposes key design parameters. Using OrchSLM as a probe, the authors analyze how orchestration behavior emerges from task structure, model-pool composition, and multi-agent consensus, motivated by constraints of SLMs versus cloud-scale LLMs.
Allow Google Analytics to measure visits? It stays off until you choose. You can change your choice at any time.