NVIDIA launches open-source "NVIDIA Recovery" to automate distributed training recovery
NVIDIA announced NVIDIA Recovery, an open-source workload-level service in the DSX OS portfolio that detects training interruptions via application logs, GPU metrics, and scheduler signals and runs a four-stage loop to attribute failures, isolate nodes, and restart or stop jobs. The service integrates with Slurm via a Helm-deployable Advisor Controller, starts in monitor mode by default, provides an audit trail and Recovery Console, and the GitHub repo and quickstart allow local deployment without physical GPUs; production monitoring reported ~8,550 anomaly detections over 3 months with mean detection and recovery-action latencies of 69s and 94s respectively.