RESEARCH · RESEARCH · #2698
WAKMan — workload-aware KV-cache manager for LLM servers
WAKMan is a workload-aware KV-cache manager (arXiv:2610.36355v1) that selects an eviction policy from live traffic without inspecting prompts or knowing request types to improve prefix-caching hit rates for LLM servers. Evaluated across mixtures of single-turn, multi-turn and agentic traffic, WAKMan raises hit rate over LRU by about 5 percentage points on average and achieves the lowest mean and worst-case optimality gap among tested policies.
KEY POINTS
- WAKMan is a workload-aware KV-cache manager (arXiv:2610.36355v1) that selects an eviction policy from live traffic without inspecting prompts or knowing request types to improve prefix-caching hit rates for LLM servers.
- Evaluated across mixtures of single-turn, multi-turn and agentic traffic, WAKMan raises hit rate over LRU by about 5 percentage points on average and achieves the lowest mean and worst-case optimality gap among tested policies.
- Improving KV-cache eviction for diverse LLM workloads can cut latency and serving cost; WAKMan offers an automated, live-traffic approach that outperforms fixed policies like LRU on average.
WHY IT MATTERS
Improving KV-cache eviction for diverse LLM workloads can cut latency and serving cost; WAKMan offers an automated, live-traffic approach that outperforms fixed policies like LRU on average.