Slurm

2026-11-02

COMPANIES · 1 SOURCE · NVIDIA Developer

NVIDIA launches open-source "NVIDIA Recovery" to automate distributed training recovery

NVIDIA announced NVIDIA Recovery, an open-source workload-level service in the DSX OS portfolio that detects training interruptions via application logs, GPU metrics, and scheduler signals and runs a four-stage loop to attribute failures, isolate nodes, and restart or stop jobs. The service integrates with Slurm via a Helm-deployable Advisor Controller, starts in monitor mode by default, provides an audit trail and Recovery Console, and the GitHub repo and quickstart allow local deployment without physical GPUs; production monitoring reported ~8,550 anomaly detections over 3 months with mean detection and recovery-action latencies of 69s and 94s respectively.

7.0

2026-10-26

COMPANIES · 1 SOURCE · NVIDIA Developer

NVIDIA Mission Control 2.5 adds autonomous recovery for Kubernetes workloads via Run:ai

NVIDIA Mission Control 2.5 extends its autonomous recovery engine — which detects, diagnoses, and recovers from failures — to Kubernetes-based AI workloads managed by NVIDIA Run:ai. The release introduces a three-layer control loop coordinating the Kubernetes scheduler, job recovery, and hardware recovery services, and notes additional features such as AI-enabled network anomaly detection with documentation and release notes available for upgrading.

6.0

2026-10-16

RESEARCH · 1 SOURCE · arXiv cs.AI

Deep RL for HPC job scheduling — statistical evaluation finds MaskablePPO best on heterogeneous trace

This arXiv preprint compares six deep reinforcement learning (DRL) treatments (PPO, A2C, DQN, each with and without action masking) against three heuristics and a seed-paired masked uniform random control on two real Slurm workload traces (homogeneous and heterogeneous) using a non‑parametric framework. Masked variants completed evaluation while unmasked ones were truncated; on the homogeneous trace no masked DRL outperformed the masked random control, whereas on the heterogeneous trace MaskablePPO improved average waiting time, average bounded slowdown, and maximum bounded slowdown (max slowdown 22% lower than LCFS), had the best average waiting-time rank among complete treatments, the lowest mean training time among masked variants, and MaskableDQN showed lowest decision latency.

5.0

2026-10-08

COMPANIES · 1 SOURCE · AWS Machine Learning

Amazon outlines multi-tenant reference architecture for SageMaker HyperPod on EKS

AWS describes a reference architecture for using Amazon SageMaker HyperPod with Amazon EKS to let multiple teams share GPU clusters while preserving isolation, fair resource quotas, and per-team cost allocation. The design uses AWS IAM Identity Center for authentication, per-team SageMaker AI domains and Kubernetes namespaces for isolation, HyperPod Task Governance for quotas, and HyperPod Observability for monitoring.

6.0

2026-10-06

COMPANIES · 1 SOURCE · AWS Machine Learning

Manage Amazon SageMaker HyperPod Spaces directly from SageMaker Studio

AWS added UI support in SageMaker Studio to create and manage SageMaker Spaces on HyperPod EKS clusters, enabling data scientists to launch JupyterLab or Code Editor environments and start/stop Spaces without using the CLI. Administrators must install the SageMaker Spaces add-on on HyperPod clusters, attach required IAM policies, and enable per-user identity propagation for Studio domains.

6.0

2026-09-22

CODING · 1 SOURCE · NVIDIA Developer

NVIDIA announces Topograph toolkit to publish cluster topology for topology-aware scheduling

NVIDIA Topograph is an open-source toolkit that discovers and normalizes cluster topology from cloud APIs and on‑prem fabrics, then publishes that model as Kubernetes node labels, Slurm configuration, or Slinky ConfigMaps so workload managers can perform topology-aware scheduling. It supports several cloud providers and on‑prem fabrics (InfiniBand, Spectrum‑X, Multi‑Node NVLink), integrates with DSX OS components such as DRA and KAI Scheduler, and is available from the dsx-ai-factory/topograph GitHub repo with Helm and native Slurm deployment options and simulation utilities.

6.0

2026-09-15

CODING · 1 SOURCE · NVIDIA Developer

Scaling federated learning across Docker, Kubernetes, and Slurm with NVIDIA FLARE

An NVIDIA Developer article describes how NVIDIA FLARE can be used to run and scale federated learning workloads across common orchestration environments — Docker, Kubernetes, and Slurm — offering guidance for moving beyond single-server, few-client setups to larger deployments.

5.0