NEWS · CODING · #3463
Amazon SageMaker HyperPod: three PyTorch DDP checks to detect and recover from silent data corruption
Amazon describes adding three integrity checks to PyTorch DistributedDataParallel training loops running on SageMaker HyperPod to detect silent data corruption (SDC): a non-finite gradient check, a ring replay audit that recomputes micro-batches across replicas for bitwise loss/gradient comparison, and a post-update weight fingerprint comparison. When a check flags a suspect rank, the system maps the rank to a node on EKS, contains the node, and uses HyperPod node replacement and checkpoints to recover; the team tested the approach with software-injected faults on clusters of ml.p5.48xlarge (32 H100 GPUs) and published code in a companion GitHub repo.
KEY POINTS
- Amazon describes adding three integrity checks to PyTorch DistributedDataParallel training loops running on SageMaker HyperPod to detect silent data corruption (SDC): a non-finite gradient check, a ring replay audit that recomputes micro-batches across replicas for bitwise loss/gradient comparison, and a post-update weight fingerprint comparison.
- When a check flags a suspect rank, the system maps the rank to a node on EKS, contains the node, and uses HyperPod node replacement and checkpoints to recover; the team tested the approach with software-injected faults on clusters of ml.p5.48xlarge (32 H100 GPUs) and published code in a companion GitHub repo.
- SDC can silently corrupt large distributed training runs and waste vast compute; these checks let training jobs detect, localize, and trigger HyperPod node replacement to limit wasted work.
WHY IT MATTERS
SDC can silently corrupt large distributed training runs and waste vast compute; these checks let training jobs detect, localize, and trigger HyperPod node replacement to limit wasted work.
SOURCES & TIMELINE
1Silent data corruption (SDC) happens when hardware, such as a GPU, returns a wrong but plausible number and raises no error. In distributed training, the wrong number can flow into the gradients and weights of every replica. Loss can still look normal for hours, and by the time a team notices an unexplained divergence or a weaker model, the compute is spent. It happens at scale: during a 54-day period of Llama 3 405B…
Foundation models deployed on Amazon SageMaker HyperPod meet inputs that no pre-curated dataset covers: domain-specific jargon, unexpected formats, new products, and changing user behavior. Without a record of production inference data, teams can’t detect drift, find failure patterns, or use real interactions to improve the model. Amazon SageMaker HyperPod now provides Data Capture, a built-in capability that record…