Tech Meridian ← ENTITY INDEX
PROMY MERIDIAN RU

TOPIC · ENTITY #33804

training operator

Related event timeline, sources and context from the news index.

EVENT TIMELINE

1

CODING · 1 SOURCE · AWS Machine Learning

Amazon SageMaker HyperPod: three PyTorch DDP checks to detect and recover from silent data corruption

Amazon describes adding three integrity checks to PyTorch DistributedDataParallel training loops running on SageMaker HyperPod to detect silent data corruption (SDC): a non-finite gradient check, a ring replay audit that recomputes micro-batches across replicas for bitwise loss/gradient comparison, and a post-update weight fingerprint comparison. When a check flags a suspect rank, the system maps the rank to a node on EKS, contains the node, and uses HyperPod node replacement and checkpoints to recover; the team tested the approach with software-injected faults on clusters of ml.p5.48xlarge (32 H100 GPUs) and published code in a companion GitHub repo.

7.0