KV cache

2026-10-28

RESEARCH · 1 SOURCE · arXiv cs.AI

WAKMan — workload-aware KV-cache manager for LLM servers

WAKMan is a workload-aware KV-cache manager (arXiv:2610.36355v1) that selects an eviction policy from live traffic without inspecting prompts or knowing request types to improve prefix-caching hit rates for LLM servers. Evaluated across mixtures of single-turn, multi-turn and agentic traffic, WAKMan raises hit rate over LRU by about 5 percentage points on average and achieves the lowest mean and worst-case optimality gap among tested policies.

6.0

2026-10-27

RESEARCH · 1 SOURCE · InfoQ AI, ML & Data Engineering

Momento CEO Khawaja Shams presents 'Serving LLMs at Scale: The Hidden KV Cache Advantage' at QCon AI

Khawaja Shams (Co‑Founder & CEO of Momento) gave a QCon AI presentation describing distributed-systems approaches to scale LLM inference efficiently, focusing on GPU utilization, KV cache memory cliffs, tiered storage, tail-latency reduction, and disaggregated prefill/decode architectures. He framed these techniques as ways to improve fleet performance, cost economics, and availability for production LLM deployments.

5.0

2026-10-22

CODING · 1 SOURCE · MIT Technology Review AI

Designing an AI serving harness built to last

The piece argues that the next competitive advantage in enterprise AI lies less in picking the best model and more in building the serving harness — the surrounding engineering (architecture, memory management, scheduling, optimization) that turns models into reliable, cost-effective production capabilities. It urges leaders to design serving architectures around workload constraints (latency, concurrency, GPU memory, cost), measure bottlenecks before optimizing, and prioritize adaptability so platforms can absorb model, hardware, and techniques changes without full rewrites.

6.0

2026-10-21

RESEARCH · 1 SOURCE · Hugging Face

VocabX tokenizer specialization cuts inference cost 77% for Qwen3-4B essay grader

Researchers/engineers report applying a method called VocabX to rebuild the tokenizer for a Qwen3-4B-based automated Portuguese essay grader while keeping the same base model, data, and training recipe; they measure a 77% reduction in inference cost by optimizing the vocabulary around per-token decode cost rather than token frequency. The numbers come from a paper currently under review and may change before publication.

7.0

2026-10-19

CODING · 1 SOURCE · AWS Machine Learning

vLLM adds bi-directional KV transfer to reduce TTFT for multi-turn agent workloads

vLLM implemented bi-directional key-value (KV) cache transfer so decode nodes can return cached KV blocks to prefill nodes across turns, avoiding repeated recomputation of earlier responses and context. In 1P1D Amazon EC2 P5en+EFA benchmarks the change reduced time-to-first-token (TTFT) by up to 1.6x for Qwen3-32B and up to 3.3x for zai-org/GLM-5-FP8; results vary by workload and config.

7.0

2026-09-30

MODELS · 1 SOURCE · NVIDIA Developer

NVIDIA publishes Dynamo-Triton HSTU recsys-examples for production inference

NVIDIA's recsys-examples repository provides an end-to-end HSTU generative recommender inference workflow that combines Dynamo-Triton, PyTorch AOTI, FlexKV-backed KV caching, and NV Embedding Cache. On an RTX PRO 6000 Blackwell GPU, the workflow's KV-cache-backed AOTI deployment achieved up to 5.93x lower latency (batch size 8, 100% GPU KV-cache hit) versus the same AOTI configuration without caching, and the guide shows export, validation (Python and native C++), and serving steps.

7.0

2026-09-10

COMPANIES · 1 SOURCE · AWS Machine Learning

Amazon SageMaker Inference adds prefix-aware routing to reduce LLM latency

Amazon SageMaker Inference now offers prefix-aware routing, a strategy that sends requests sharing the same prompt prefix to the same instance so the KV cache stays warm. In benchmarks on Llama 3.1 70B, this reduced P50 time-to-first-token by up to 77% and increased KV cache hit rates from about 25% to over 80%.

7.0

2026-09-10

MODELS · 1 SOURCE · DeepSeek

DeepSeek launches DeepSeek‑V4.1‑Flash with new causal encoder–decoder and lower costs

DeepSeek announced DeepSeek‑V4.1‑Flash, a smaller-native-vision model using a new Causal Encoder–Decoder design (8B active params for input, 16B for output), new pretraining and larger-scale RL post-training, plus KV‑cache compression. The company is retiring V4‑Flash variants, routing deepseek-v4-pro traffic to V4.1‑Flash from 04:00 UTC on Sept 14, 2026, and changing pricing (new rates take effect 04:00 UTC on Sept 10, 2026) while partners WorkBuddy, CodeBuddy and OpenCode already support V4.1‑Flash.

8.0