PARTNERSHIP · CODING · #2935
Google and Meta open-source TorchTPU to provide native PyTorch backend for Google TPUs
Google, in collaboration with Meta, open-sourced TorchTPU, a native PyTorch backend for Google TPUs that prioritizes eager execution while also supporting compiled modes, distributed training, and hardware optimizations; migrating common PyTorch code generally requires only swapping cuda->tpu and nccl->tpu_dist or using torch.accelerator. Google validated TorchTPU across ~5,000 Hugging Face models and reported single-TPUv7 inference of 25.7k token/s (seq_len 5120) and 22.4k token/s (seq_len 10240) for a 30B Muse Glimmer model, plus multi-chip training numbers and large-scale MoE experiments up to 8,192 TPUv7 chips.
KEY POINTS
- Google, in collaboration with Meta, open-sourced TorchTPU, a native PyTorch backend for Google TPUs that prioritizes eager execution while also supporting compiled modes, distributed training, and hardware optimizations; migrating common PyTorch code generally requires only swapping cuda->tpu and nccl->tpu_dist or using torch.accelerator.
- Google validated TorchTPU across ~5,000 Hugging Face models and reported single-TPUv7 inference of 25.7k token/s (seq_len 5120) and 22.4k token/s (seq_len 10240) for a 30B Muse Glimmer model, plus multi-chip training numbers and large-scale MoE experiments up to 8,192 TPUv7 chips.
- TorchTPU lowers friction for running standard PyTorch workloads on TPU hardware with minimal code changes and could shift high-performance PyTorch development and cost trade-offs toward Google TPUs.
WHY IT MATTERS
TorchTPU lowers friction for running standard PyTorch workloads on TPU hardware with minimal code changes and could shift high-performance PyTorch development and cost trade-offs toward Google TPUs.