WAKMan — workload-aware KV-cache manager for LLM servers
WAKMan is a workload-aware KV-cache manager (arXiv:2610.36355v1) that selects an eviction policy from live traffic without inspecting prompts or knowing request types to improve prefix-caching hit rates for LLM servers. Evaluated across mixtures of single-turn, multi-turn and agentic traffic, WAKMan raises hit rate over LRU by about 5 percentage points on average and achieves the lowest mean and worst-case optimality gap among tested policies.