Validate GPU Cluster Readiness Before AI Workloads Land

extract

A GPU cluster can pass every health check and still fail to run an AI workload. Even when every GPU, network link, and pod reports healthy, a 512-GPU training job can underperform or fail. The cause may be one slow GPU, a link that degrades under load, or a configuration that quietly routes traffic over a slower path. Operators may not discover the problem until hours into the run or until a customer files a ticket. Teams can then spend days bisecting the cluster to find the root cause while the capacity sits idle. NVIDIA Cluster Readiness Engine (NVCRE) is an open source Kubernetes controller that narrows the search to the specific nodes involved before production workloads land. It runs real distributed workloads across topology-aware node groups, measures the results, and reports which nodes failed each test. Operators no longer need to write NVIDIA Collective Communications Library (NCCL) manifests by hand, bisect racks manually, or learn about degraded hardware from customer tickets. Readiness becomes a proven property of the cluster rather than an assumption. What does proving readiness require A GPU cluster becomes ready in stages. It moves through bring-up, burn-in, preproduction, and production, with each stage setting a different bar. A node that passes a smoke test is not necessarily ready to join a 512-GPU training run. Platform teams often encode that progression in a runbook, spreadsheet, or shell scripts wrapped around NCCL tests. It becomes another system they must build and maintain alongside node configuration, GPU sharing, and workload orchestration. A cluster can pass standard diagnostics and still fail under a real distributed job, so the best way to test readiness is to run a workload. On Slurm, that requires a single srun command. Kubernetes has no built-in equivalent, so the same test requires GPU and remote direct memory access (RDMA) resource requests, NCCL settings matched to the network fabric, a large enough shared-memory volume, and a way to ensure that all pods start together. NVCRE fills these gaps on Kubernetes. It runs workloads that expose real hardware problems and names exactly which node caused each failure. communication/nccl-all-reduce Status: Failed Runtime: 42m 18s Scale: full-scale Nodes/Job: 8 Failed Nodes: gpu-node-07 ThresholdViolation gpu-node-12 HardwareFailureDetected Summary Categories: 1/2 passed Failed Nodes: 2 Result: FAILED How NVCRE works The API is the product surface. Custom resource definitions (CRDs) define each resource, so you can inspect it with kubectl and manage it through GitOps workflows. A layered API The API has three resources arranged in a hierarchy. - Certification: The resource you create. It names the nodes to test and the categories to run. - Workflow: Manages one category. It applies catalog, platform, and GPU overrides; manages iteration count; sets the orchestration target; and creates the child job. - Job: Runs the workload for the target node group, monitors node health, and records measurements and failures. A certification creates one workflow per category, and each workflow creates its child job. Results then propagate upward. The job records which nodes failed and why, the workflow reports the test result, and the certification groups results by category. That hierarchy attributes each failure to a specific node and category. For example, a run reports that gpu-01 hit a hardware fault during NCCL and that gpu-02 missed its bandwidth target. Validating a cluster The following example shows how a certification names its targets and the categories to run. apiVersion: nvcre.nvidia.com/v1alpha1 kind: Certification metadata: name: gpu-cluster-cert spec: target: nodeSelector: nvidia.com/gpu.present: "true" categories: - domain: communication variant: nccl-all-reduce - domain: training variant: nemotron5-8b $ kubectl apply -f certification.yaml $ kubectl get certifications.nvcre.nvidia.com -w The built-in catalog currently covers three domains: five NCCL communication variants (all-reduce, all-gather, all-to-all, loopback, and loopback across NVIDIA NVSwitch), the NVIDIA Data Center GPU Manager (DCGM) level-4 diagnostic suite, and NVIDIA NeMo pretraining with NVIDIA Nemotron 5 models at 8B and 56B parameters. Each entry includes platform-aware defaults. NVCRE detects the GPU architecture and cloud platform from the target nodes and derives the rest of the configuration, including GPUs per node, the NCCL environment, and platform-specific networking. Pass criteria as expressions Pass and fail criteria use Common Expression Language (CEL) and are evaluated against measured metrics. No thresholds ship by default. The values below are illustrative examples for NVIDIA GB200 NVL72-class systems. categories: - domain: communication variant: nccl-all-reduce options: thresholds: busBandwidthGBps: "value >= 900" - domain: training variant: nemotron5-8b options: thresholds: goodputRatio: "value >= 0.9" avgTFLOPsPerGPU: "value >= 800" When a measured metric misses its target, NVCRE sets a ValidationFailed condition, recorded separately from whether the run itself succeeded. A workload that finishes but misses its target is still reported as a failure. Testing at the scale where failures appear Some failures are visible only at a particular scale, so the grouping strategy is explicit. The testScale field selects the strategy. - Intra-node tests each node independently. - Intra-rack partitions nodes by topology domain using the nvidia.com/gpu.clique label. - Full-scale puts every node in a single group. - Diagnose runs adaptive fault isolation (covered next). The strategy you select dictates what gets measured. An NCCL test inside one NVIDIA NVLink domain measures NVLink bandwidth, while the same test across three racks measures the scale-out fabric. The two measure different things. Adaptive fault isolation The hardest case in multi-node validation is a failure that cannot be attributed to any single node. A 64-node all-reduce returns low bandwidth, and every node in the group is equally implicated. Isolating the cause by hand can take days of engineering time. NVCRE automates that isolation. Setting testScale: diagnose runs topology-aware hierarchical group testing. The engine splits each failing group, reruns the halves, and continues until it reaches minGroupSize. Groups that still fail at that size are flagged as suspects. maxConcurrent limits the number of jobs that run concurrently so they do not saturate the fabric being measured. The output names a small number of suspect nodes instead of implicating the entire group and gives the reason each node failed. Running any workload: the WorkloadRun API Running a multi-node GPU workload on Kubernetes requires platform detection, framework-specific runtime configuration, GPU and network resource requests, and cleanup after a failed run. That setup is repetitive and error-prone. WorkloadRun handles the setup: provide a container image, select a framework, and specify the number of nodes. apiVersion: nvcre.nvidia.com/v1alpha1 kind: WorkloadRun metadata: name: nccl-all-reduce spec: image: nvcr.io/nvidia/pytorch:26.01-py3 framework: mpi: binary: /usr/local/bin/all_reduce_perf_mpi args: ["-b", "8", "-e", "32G", "-f", "2", "-n", "100"] mpirunPath: /usr/local/mpi/bin/mpirun numNodes: 4 bandwidthMeasurement: logProfileRef: nccl-bandwidth testType: all_reduce The framework field supports exactly one of torch (distributed training through torchrun), mpi (NCCL tests and other MPI workloads), or exec (an arbitrary command). NVCRE generates the matching Kubeflow TrainingRuntime, injects the shared-memory volume, sets the NCCL and platform environment variables, and enables NVIDIA NVLink scale-up networking where the hardware supports it. Without a gang scheduler, the default Kubernetes scheduler places pods independently, and ranks wait for their peers at the framework rendezvous. On a busy cluster, that can deadlock: partially placed pods hold GPUs while waiting for peers that never arrive. Setting spec.gangScheduler opts every workload pod into a gang-aware scheduler, such as KAI Scheduler, which holds all pods until the entire gang can be placed at once. Because WorkloadRun is a plain CRD, external tools can use it to run workloads without adopting the rest of the NVCRE. NVCRE provides the execution path, and the calling tool supplies the test. Configure, validate, and monitor AI clusters at scale A cluster must answer three questions on the way to production: Is it configured correctly? Is it ready to run a real AI workload? Is it healthy right now? A different layer of NVIDIA DSX OS addresses each question. NVIDIA AI Cluster Runtime (AICR) establishes and maintains a validated cluster configuration. AICR captures validated combinations of drivers, operators, kernels, and system settings as version-locked recipes. Teams can reproduce the same optimized configuration across clusters, validate live state, and detect drift. This reduces performance variation and avoids days of tuning while costly GPUs sit idle. NVCRE verifies that a cluster is ready to run real AI workloads. This active, workload-driven layer generates load, so it can find failures that produce no telemetry. A single degraded GPU slows a synchronous training job to the speed of its worst rank, and an NCCL bandwidth test can identify the problem in minutes. NVIDIA NVSentinel continuously monitors cluster health. As the passive, telemetry-driven layer, it watches signals the cluster already produces, including DCGM metrics, Xid errors, system logs, and cloud provider maintenance events. It detects runtime faults and can drive quarantine, drain, and remediation workflows. Because it consumes no GPU time, it can run continuously in production, where active testing would take GPUs away from workloads. Each project delivers value independently and integrates with the others for teams running the full stack. NVCRE records failed nodes and reasons. It does not cordon, taint, or patch node conditions, avoiding conflicting actions and desynchronized cleanup. The NVSentinel NVCRE Certification Monitor can translate failed certification results into health events. Configured NVSentinel policies can then quarantine and drain nodes or trigger external remediation. A later successful certification can clear the failure signal and release the taint. Get started NVCRE requires Kubernetes 1.29 or later, kubectl, Helm 3.x, and NVIDIA GPU Operator on the target cluster. NVIDIA GB200 NVL72 and NVIDIA GB300 NVL72 catalog entries also require the NVIDIA DRA Driver for GPUs because those entries create ComputeDomain resources. The DCGM level-4 category requires the standalone DCGM service. A gang-aware scheduler such as KAI Scheduler is optional but recommended on busy, shared clusters. Install the CLI and set up the cluster. The nvcrectl setup init command installs the CRDs, controller, Kubeflow Trainer, and default log profiles. $ curl -fsSL https://github.com/NVIDIA/cluster-readiness-engine/releases/latest/download/installer | bash $ nvcrectl setup init Then run an end-to-end validation. $ nvcrectl certification run --cert-file certification.yaml --wait $ nvcrectl certification report gpu-cluster-cert -n <namespace> The following report lists each category’s status, runtime, measured bandwidth, and any failed nodes, with the reason for each failure. Certification Report Name: gpu-cluster-cert Platform: aws GPU: gb300 Nodes: 16 communication/nccl-all-reduce Status: Succeeded Runtime: 3m 56s Scale: full-scale Nodes/Job: 16 MNNVL: Enabled Bandwidth: Size AlgBW BusBW Samples 16 GB 473.44 GB/s 932.09 GB/s 9 Summary Categories: 1/1 passed Failed Nodes: none Result: PASSED Get involved NVCRE is licensed under Apache 2.0 and developed in the open. Report a bug, request a feature, or propose a change before sending a pull request on GitHub. You can also contribute catalog entries, workload adapters, tests, and documentation. Use the engine to validate GPU clusters before production. Its shared workflow and test catalog run consistently across Kubernetes clusters, with roadmap support for new NVIDIA architectures, inference, and automated lifecycle validation. Validate your GPU cluster before your next production workload lands. The project is part of NVIDIA DSX OS, the operating layer of the NVIDIA DSX AI Factory Platform.