NEWS · RESEARCH · #366
Where Should the KV Cache Live? Placement policies across GPU, CPU, and SSD
This arXiv preprint simulates KV-cache tiering across GPU HBM, CPU DRAM, and SSD for chat, agent loops, and document QA workloads, calibrated with a random-forest execution-time predictor. It finds tiering enables ~73.02× more concurrent sessions per GPU and ~62.04× lower cost per session, placement mainly affects PCIe migration traffic and time-to-first-token (decode is compute-bound at batch size 1), recency works well for chat while reuse-frequency is best for agents and document QA, the paper finds the existing "predicted reuse" implementation is byte-identical to recency, EWMA differs but still lags reuse-frequency, and prefetching does not justify its bandwidth cost.
KEY POINTS
- This arXiv preprint simulates KV-cache tiering across GPU HBM, CPU DRAM, and SSD for chat, agent loops, and document QA workloads, calibrated with a random-forest execution-time predictor.
- It finds tiering enables ~73.02× more concurrent sessions per GPU and ~62.04× lower cost per session, placement mainly affects PCIe migration traffic and time-to-first-token (decode is compute-bound at batch size 1), recency works well for chat while reuse-frequency is best for agents and document QA, the paper finds the existing "predicted reuse" implementation is byte-identical to recency, EWMA differs but still lags reuse-frequency, and prefetching does not justify its bandwidth cost.
- The findings guide practical cache-tiering and placement choices for large-scale LLM serving, showing big capacity/cost gains from tiering and overturning some existing predicted-reuse and prefetch recommendations.
WHY IT MATTERS
The findings guide practical cache-tiering and placement choices for large-scale LLM serving, showing big capacity/cost gains from tiering and overturning some existing predicted-reuse and prefetch recommendations.