Tech Meridian ← LIVE FEED
PROMY MERIDIAN RU

RESEARCH · RESEARCH · #3187

FlashOffloading: KV-cache offloading architecture for long-context LLM serving (vLLM‑Ascend v0.30.0)

The paper presents FlashOffloading, a host‑DRAM‑centric architecture that fully offloads the KV cache for both prefill and decode phases and adds a rolling HBM cache, hot decode cache, reverse sparse index, and compute–load overlap to minimize offloading latency. In experiments on an Ascend910C PD‑disaggregated platform running GLM‑5.2 with MTP3 speculative decoding, FlashOffloading improved prefill throughput up to 1.38× and decode throughput up to 1.91× under a 30 ms time‑per‑output‑token (TPOT) constraint; the design is integrated into the vLLM‑Ascend open‑source release v0.30.0.

KEY POINTS

  1. The paper presents FlashOffloading, a host‑DRAM‑centric architecture that fully offloads the KV cache for both prefill and decode phases and adds a rolling HBM cache, hot decode cache, reverse sparse index, and compute–load overlap to minimize offloading latency.
  2. In experiments on an Ascend910C PD‑disaggregated platform running GLM‑5.2 with MTP3 speculative decoding, FlashOffloading improved prefill throughput up to 1.38× and decode throughput up to 1.91× under a 30 ms time‑per‑output‑token (TPOT) constraint; the design is integrated into the vLLM‑Ascend open‑source release v0.30.0.
  3. By reducing KV‑cache HBM pressure while keeping host‑to‑device latency low, FlashOffloading enables higher effective concurrency and throughput for long‑context LLM inference and is already available in an open‑source serving framework.

WHY IT MATTERS

By reducing KV‑cache HBM pressure while keeping host‑to‑device latency low, FlashOffloading enables higher effective concurrency and throughput for long‑context LLM inference and is already available in an open‑source serving framework.

SOURCES & TIMELINE

1