RELEASE · COMPANIES · #2957
NVIDIA launches open-source "NVIDIA Recovery" to automate distributed training recovery
NVIDIA announced NVIDIA Recovery, an open-source workload-level service in the DSX OS portfolio that detects training interruptions via application logs, GPU metrics, and scheduler signals and runs a four-stage loop to attribute failures, isolate nodes, and restart or stop jobs. The service integrates with Slurm via a Helm-deployable Advisor Controller, starts in monitor mode by default, provides an audit trail and Recovery Console, and the GitHub repo and quickstart allow local deployment without physical GPUs; production monitoring reported ~8,550 anomaly detections over 3 months with mean detection and recovery-action latencies of 69s and 94s respectively.
KEY POINTS
- NVIDIA announced NVIDIA Recovery, an open-source workload-level service in the DSX OS portfolio that detects training interruptions via application logs, GPU metrics, and scheduler signals and runs a four-stage loop to attribute failures, isolate nodes, and restart or stop jobs.
- The service integrates with Slurm via a Helm-deployable Advisor Controller, starts in monitor mode by default, provides an audit trail and Recovery Console, and the GitHub repo and quickstart allow local deployment without physical GPUs; production monitoring reported ~8,550 anomaly detections over 3 months with mean detection and recovery-action latencies of 69s and 94s respectively.
- Automating workload-level recovery reduces manual intervention and wasted GPU-hours, improving training goodput and lowering operational overhead for large-scale distributed AI training.
WHY IT MATTERS
Automating workload-level recovery reduces manual intervention and wasted GPU-hours, improving training goodput and lowering operational overhead for large-scale distributed AI training.
SOURCES & TIMELINE
1NVIDIA Recovery automates workload-level recovery across a cluster's training jobs around the clock, detecting interruptions through application logs, GPU metrics, and scheduler signals. The service coordinates a four-stage recovery loop that detects anomalies, attributes failures, isolates implicated nodes, and restarts or stops workloads based on configured policy. Integration with Slurm uses a reference Advisor …
Use this open-source tool to correlate workload-scoped logs, metrics, and scheduler state into a reproducible diagnosis on your own cluster. NVIDIA Attribution correlates workload-scoped logs, metrics, telemetry, and scheduler state to produce a structured diagnosis with outcome, root cause, affected nodes, supporting symptoms, and an evidence trail. The system operates through two deployable components: the Attrib…