NCCL

2026-11-02

COMPANIES · 1 SOURCE · NVIDIA Developer

NVIDIA launches open-source "NVIDIA Recovery" to automate distributed training recovery

NVIDIA announced NVIDIA Recovery, an open-source workload-level service in the DSX OS portfolio that detects training interruptions via application logs, GPU metrics, and scheduler signals and runs a four-stage loop to attribute failures, isolate nodes, and restart or stop jobs. The service integrates with Slurm via a Helm-deployable Advisor Controller, starts in monitor mode by default, provides an audit trail and Recovery Console, and the GitHub repo and quickstart allow local deployment without physical GPUs; production monitoring reported ~8,550 anomaly detections over 3 months with mean detection and recovery-action latencies of 69s and 94s respectively.

7.0

2026-10-29

MODELS · 1 SOURCE · AWS Machine Learning

NVIDIA publishes Nemotron 3.5 Super VL (120B MoE) and provides AWS EKS / SageMaker deployment guide

NVIDIA released Nemotron 3.5 Super VL, an open multimodal 120B-parameter mixture-of-experts (MoE) model that activates about 12B parameters per token and is distilled from Nemotron 3 Ultra. NVIDIA published the model weights, training datasets, and post-training recipes and published reference implementations and an aws-do-eks walkthrough showing how to deploy Nemotron models on Amazon EKS, EKS Auto Mode, and SageMaker HyperPod EKS; the model is validated on AWS GPU instances and integrates with major agent frameworks via the NVIDIA Agent Toolkit.

7.0

2026-10-28

CODING · 1 SOURCE · AWS Machine Learning

AWS Ray Train DLC (train-ml-cuda-v1.1) enables EFA-backed multi-node PyTorch on EKS

AWS provides a Ray Train Deep Learning Container (public.ecr.aws/deep-learning-containers/ray:train-ml-cuda-v1.1) that bundles CUDA, NCCL, EFA userspace libraries and Ray, allowing multi-node PyTorch training on EKS without building custom images. A provided sample shows fine-tuning Qwen2.5-1.5B with PyTorch FSDP across eight NVIDIA L4 GPUs on two g6.12xlarge instances using KubeRay and EFA.

6.0

2026-10-22

COMPANIES · 1 SOURCE · NVIDIA Developer

NVIDIA adds official eight-node DGX Spark clusters with 1 TB unified memory to run frontier models

At GTC Berlin NVIDIA announced official support for eight-node DGX Spark clusters, scaling unified memory to 1 TB and enabling local execution of frontier models including the full GLM 5.3 (753B) from Z.ai. NVIDIA also introduced a Spark-optimized collective backend called RoCEnante (merged into FlashInfer) that reduces all-reduce latency—reporting a 4.43x decode speedup at eight nodes (vs 3.05x without it)—and published benchmarks (e.g., Qwen3.8-27B NVFP4 scaling from ~12 to 52 tokens/sec, GLM 5.3 ~80 TPS, and 7.79x pretraining speedup at 97.4% efficiency for a nanochat model); NeMo Switchyard and vLLM integration are highlighted for multi-model routing and distributed inference, and NVIDIA Sync eight-node support is due later this year.

7.0

2026-10-15

RESEARCH · 1 SOURCE · arXiv cs.AI

Echo: error‑compensated 4‑bit all-reduce for tensor‑parallel LLM training and inference

The arXiv paper introduces Echo, a 4‑bit all‑reduce protocol for tensor-parallel LLM training and inference that moves 4‑bit data but sums in high precision and feeds per-call rounding errors forward so they cancel rather than accumulate. The authors prove the accumulated error is depth‑independent and report that Echo is 2.09× faster than 16‑bit NCCL all‑reduce on 1 GB messages on A100s, speeds time‑to‑first‑token and training step latency by up to ~1.16×/1.09× on idle interconnects (and larger gains under background traffic), keeps average accuracy within 0.4 points on Qwen3‑8B, and yields <0.9% higher final loss after 10B‑token pretraining — with no FP8/FP4 hardware required.

7.0

2026-09-23

CODING · 1 SOURCE · NVIDIA Developer

NVIDIA open-sources NVCRE, a Kubernetes controller to validate GPU cluster readiness

NVIDIA Cluster Readiness Engine (NVCRE) is an open-source Kubernetes controller that runs real distributed workloads across topology-aware node groups to validate GPU cluster readiness before production AI jobs start. It exposes Certification/Workflow/Job CRDs, adaptive fault isolation to pinpoint failing nodes, a WorkloadRun API for multi-node GPU setup, integrates with NVIDIA AI Cluster Runtime and NVSentinel (part of NVIDIA DSX OS), and is installable via nvcrectl; contributors can extend tests and workload adapters on GitHub.

7.0

2026-09-21

COMPANIES · 1 SOURCE · NVIDIA Developer

NVIDIA Dynamo-Triton 26.07 enables TensorRT multi-device inference across GPUs

NVIDIA has enabled TensorRT multi-device inference in Dynamo-Triton release 26.07 (TensorRT multi-device is supported starting in TensorRT 11.0), allowing a single KIND_MODEL instance to own multiple GPUs and present one gRPC endpoint while using NCCL-backed distributed collectives. In a Cosmos 3 Nano video-generation demo, distributing a 44,160-token transformer across up to eight GPUs reduced end-to-end generation latency from 156.6s to 34.2s and produced a reported transformer RPC speedup of 6.09x; users can download Dynamo-Triton 26.07 from NGC and follow the TensorRT multi-device guide to reproduce the setup.

7.5

2026-09-16

CODING · 1 SOURCE · AWS Machine Learning

NVIDIA NVRx adds fault-tolerance to PyTorch FSDP training on Amazon EKS

NVIDIA describes how to integrate its Resiliency Extension (NVRx) with PyTorch FSDP on Amazon EKS to reduce downtime for large multi-node GPU training. The post demonstrates async checkpointing (TorchAsyncCheckpoint), in-process restart, and an in-job restart launcher (ft_launcher), provides H100 2–8 node benchmarks, and publishes reproducible code and a pip-installable nvidia-resiliency-ext package.

6.0