NVIDIA described Recovery, part of DSX OS, as an open-source service to detect interruptions and restart distributed training jobs.

NVIDIA said on Nov. 2, 2026, that distributed AI training increasingly needs automated recovery as failures across hardware and software dependencies can stall progress, idle GPUs and raise compute costs. The company framed NVIDIA Recovery, part of NVIDIA DSX OS, as an open-source, workload-level service designed to detect interruptions and coordinate restarts across a cluster’s training jobs.

In the blog post, NVIDIA said Recovery uses application logs, GPU metrics and scheduler signals to detect crashes, hangs and stragglers, then applies failure attribution to identify implicated resources and coordinate isolation and checkpoint restarts on healthy capacity. The company said the system is intended to automate recovery across workloads around the clock, reducing manual intervention and helping sustain training goodput.

The post said Recovery sits at the cluster level, distinct from infrastructure remediation and from in-job libraries such as NVRx. NVIDIA described break-fix automation as handling faulty nodes, while NVRx provides recovery, hang detection and straggler mitigation within individual jobs. Recovery, by contrast, is meant to manage cross-workload restarts, placement and node exclusion.

NVIDIA said the service integrates with Slurm and can work alongside other components in its health and resilience stack. The post said Kubernetes deployments may use NVSentinel to monitor GPU health metrics and XID events and automatically cordon and drain faulty nodes, while many Slurm sites rely on custom prolog and epilog scripts. NVIDIA said the slurm-operator from the Slinky project natively supports and injects those scripts when running Slurm on Kubernetes.

Source link