RELEASE · CODING · #3470
Google open-sources 'TPU Sync' C++ library to move KV cache and weights on TPUs
Google Cloud released TPU Sync, a native C++ library (Apache 2.0) that moves KV cache blocks and model weights between TPU HBM, host DRAM, shared storage, and the network, with Python wheels for JAX and PyTorch. Google reports large latency and TTFT reductions for long-context and disaggregated LLM serving (e.g., Qwen3 235B median KV transfer latency down to ~230 ms and average TTFT reductions of ~60%, up to 90% for a 480B model), but several features (storage tier, RL weight sync) and Python packages are experimental or development builds and APIs may change rapidly.
KEY POINTS
- Google Cloud released TPU Sync, a native C++ library (Apache 2.0) that moves KV cache blocks and model weights between TPU HBM, host DRAM, shared storage, and the network, with Python wheels for JAX and PyTorch.
- Google reports large latency and TTFT reductions for long-context and disaggregated LLM serving (e.g., Qwen3 235B median KV transfer latency down to ~230 ms and average TTFT reductions of ~60%, up to 90% for a 480B model), but several features (storage tier, RL weight sync) and Python packages are experimental or development builds and APIs may change rapidly.
- TPU Sync targets the data-movement bottleneck that limits long‑context and disaggregated LLM serving (and adds experimental RL weight synchronization), potentially making large-context TPU serving far more practical.
WHY IT MATTERS
TPU Sync targets the data-movement bottleneck that limits long‑context and disaggregated LLM serving (and adds experimental RL weight synchronization), potentially making large-context TPU serving far more practical.