NVIDIA says Mission Control 2.5 brings autonomous recovery to Kubernetes workloads managed by Run:ai, adding job and hardware remediation controls.
NVIDIA said on October 26, 2026, that Mission Control 2.5 extends its autonomous recovery engine to Kubernetes-based workloads managed with NVIDIA Run:ai, moving a resiliency framework that was previously limited to Slurm-based high-performance computing jobs into Kubernetes environments. The company framed the update as part of a broader push toward self-healing AI factory infrastructure, where large training runs can continue with less manual intervention when hardware or software faults arise.
According to NVIDIA’s technical blog published October 26, Mission Control’s autonomous recovery engine combines two services: autonomous job recovery and autonomous hardware recovery. In the Kubernetes setup, Run:ai acts as the default scheduler, while the job-recovery service uses workload context, failure history and job topology to decide whether a workload should restart, retry, re-queue or be placed elsewhere. The hardware-recovery layer monitors nodes and GPUs, isolates anomalies from scheduling and runs diagnostics or remediation so suspect equipment does not keep circulating into new jobs.
The company described three recovery paths. In a hardware-fault scenario during training, the system can restart or reschedule workloads onto healthy nodes while the faulty node is isolated and diagnosed. For transient software faults, it can retry according to scheduler policy without quarantining healthy hardware. If degradation is detected early, it can remove a node from scheduling before a job lands on it. NVIDIA said the separation of scheduling, job context and hardware-health functions is intended to make the model portable across orchestrators.
Source link
