Tech Meridian ← LIVE FEED
PROMY MERIDIAN RU

NEWS · CODING · #3463

Amazon SageMaker HyperPod: three PyTorch DDP checks to detect and recover from silent data corruption

Amazon describes adding three integrity checks to PyTorch DistributedDataParallel training loops running on SageMaker HyperPod to detect silent data corruption (SDC): a non-finite gradient check, a ring replay audit that recomputes micro-batches across replicas for bitwise loss/gradient comparison, and a post-update weight fingerprint comparison. When a check flags a suspect rank, the system maps the rank to a node on EKS, contains the node, and uses HyperPod node replacement and checkpoints to recover; the team tested the approach with software-injected faults on clusters of ml.p5.48xlarge (32 H100 GPUs) and published code in a companion GitHub repo.

KEY POINTS

  1. Amazon describes adding three integrity checks to PyTorch DistributedDataParallel training loops running on SageMaker HyperPod to detect silent data corruption (SDC): a non-finite gradient check, a ring replay audit that recomputes micro-batches across replicas for bitwise loss/gradient comparison, and a post-update weight fingerprint comparison.
  2. When a check flags a suspect rank, the system maps the rank to a node on EKS, contains the node, and uses HyperPod node replacement and checkpoints to recover; the team tested the approach with software-injected faults on clusters of ml.p5.48xlarge (32 H100 GPUs) and published code in a companion GitHub repo.
  3. SDC can silently corrupt large distributed training runs and waste vast compute; these checks let training jobs detect, localize, and trigger HyperPod node replacement to limit wasted work.

WHY IT MATTERS

SDC can silently corrupt large distributed training runs and waste vast compute; these checks let training jobs detect, localize, and trigger HyperPod node replacement to limit wasted work.

SOURCES & TIMELINE

1