NVIDIA's Cluster Readiness Engine certifies GPU clusters for production by running real AI workloads. Learn how it identifies bottlenecks and faulty nodes.

NVIDIA has launched the Cluster Readiness Engine (NVCRE), a Kubernetes-based open-source tool aimed at ensuring GPU clusters are production-ready for the demands of AI workloads. By running actual distributed jobs, such as training benchmarks and communication tests, NVCRE identifies underperforming nodes, degraded hardware, and configuration flaws that standard diagnostics may miss.

AI training jobs often fail or underperform due to subtle issues like a single slow GPU or misconfigured traffic routing. NVCRE addresses this by proactively running topology-aware tests, measuring results, and diagnosing failures before workloads go live. For example, during a 512-GPU training run, a single bottleneck often slows the entire operation. Using NVCRE, teams can pinpoint the exact node or link causing the issue, saving days of troubleshooting.

At its core, NVCRE operates as a Kubernetes controller using custom resources like Certification, Workflow, and Job. These resources organize the testing process, from defining the nodes to target, to executing distributed jobs, and reporting failures with machine-readable reasons. For example, a test might flag node gpu-node-07 for bandwidth issues or another node for hardware degradation.

The tool is designed to test clusters at scale, with configurations for intra-node, intra-rack, and full-scale testing. NVCRE’s adaptive fault isolation feature automates the process of narrowing down issues in multi-node failures, a task that could otherwise require significant engineering time.

The release of NVCRE comes at a time when the demand for large-scale AI infrastructure is surging.

Source link