GUIDE · CODING · #496
NVIDIA NVRx adds fault-tolerance to PyTorch FSDP training on Amazon EKS
NVIDIA describes how to integrate its Resiliency Extension (NVRx) with PyTorch FSDP on Amazon EKS to reduce downtime for large multi-node GPU training. The post demonstrates async checkpointing (TorchAsyncCheckpoint), in-process restart, and an in-job restart launcher (ft_launcher), provides H100 2–8 node benchmarks, and publishes reproducible code and a pip-installable nvidia-resiliency-ext package.
KEY POINTS
- NVIDIA describes how to integrate its Resiliency Extension (NVRx) with PyTorch FSDP on Amazon EKS to reduce downtime for large multi-node GPU training.
- The post demonstrates async checkpointing (TorchAsyncCheckpoint), in-process restart, and an in-job restart launcher (ft_launcher), provides H100 2–8 node benchmarks, and publishes reproducible code and a pip-installable nvidia-resiliency-ext package.
- This matters because NVRx’s layered recovery (async checkpoints, in-process restart, ft_launcher) can cut wasted GPU time and shorten recovery from soft and hard faults in large distributed PyTorch jobs on EKS.
WHY IT MATTERS
This matters because NVRx’s layered recovery (async checkpoints, in-process restart, ft_launcher) can cut wasted GPU time and shorten recovery from soft and hard faults in large distributed PyTorch jobs on EKS.