loading
loading
Training large-scale AI models on thousands of GPUs is a high-stakes operation where monitoring and automation must be built-in to optimize cluster resiliency and uptime at scale. This concept isn’t new to data center operators who have measured mean-time-to-repair (MTTR) and mean-time-between-failure (MTBF) long before AI factories existed. Now the stakes are higher. Modern AI factories run distributed training jobs across hundreds of nodes for days or weeks, and at this scale the operating model changes: hardware anomalies, software errors, network timeouts, and resource contention must be managed proactively to keep workloads on track. Traditional approaches that rely on manual intervention, static checkpointing, and reactive monitoring simply don’t scale when organizations are running dozens of these concurrent multi-node workloads in shared infrastructure. This is why software services, like the autonomous recovery engine offered in NVIDIA Mission Control, are critical to scaling production AI. Previously, the autonomous recovery engine, which can detect, diagnose, and recover from anomalies across the stack, was limited to high-performance computing (HPC) workloads leveraging Slurm. As a part of Mission Control 2.5, this service is now extended to Kubernetes-based workloads managed using NVIDIA Run:ai. This post demonstrates how this latest version of Mission Control brings its integrated resiliency framework to Kubernetes workloads, helping AI factory operations move from reactive operational maintenance to proactive, self-healing infrastructure. Built-in resiliency services The vision behind Mission Control’s autonomous recovery engine is simple: AI factories should be self-healing systems where failures are detected instantly, recovery decisions are made intelligently based on workload context and failure type, and jobs resume automatically with minimal disruption or wasted compute. The goal is to understand what failed, isolate anomalies and tag them for maintenance, and return workloads to productive compute with as little wasted time as possible. There are two services that make up the autonomous recovery engine: - autonomous job recovery - autonomous hardware recovery Because true self-governing resiliency is best understood as a control loop rather than a single recovery feature, the autonomous recovery engine then works with the workloads scheduler (HPC or Kubernetes) providing an integrated three-layer framework where each component plays a distinct but coordinated role. For this post, we’ll be focusing on scheduling Kubernetes-based workloads. The three roles in the loop Let’s start with the Kubernetes workload placement layer. In Mission Control, NVIDIA Run:ai is the Kubernetes default scheduler and included as a part of the software. It manages the Kubernetes-based AI workloads, queues, policies, and schedules the jobs onto available capacity. The autonomous job recovery service provides the workload-context layer. It watches running jobs, tracks where each job is executing, and uses failure history and job topology to classify what kind of recovery path makes sense. The resource manager integration determines how that guidance becomes a restart, retry, re-queue, or placement action. The autonomous hardware recovery service is the infrastructure-health layer. It monitors nodes and GPUs, removes anomalies from scheduling when needed, and runs diagnostics or remediation so hardware problems do not keep spreading into new jobs. Keeping these responsibilities separate is what makes the model portable. The autonomous job recovery service doesn’t need to repair hardware and the hardware recovery service doesn’t need to understand every training framework. Finally, the scheduler can steer workloads toward ready capacity without waiting for a root-cause diagnosis. Failure paths in action The following scenarios show how the control loop behaves differently in Mission Control depending on the failure signal detected by its autonomous recovery engine. Scenario 1. Hardware fault during training A distributed training job is running across a pool of GPUs when one GPU starts to fail. The symptom may appear first as ECC errors, a hung worker, or a communication timeout in the job. The important thing is what happens next: recovery splits into two parallel tracks. The autonomous job recovery service captures the job context and identifies the suspect node or GPU. The Kubernetes scheduler integration then moves workload recovery forward either by restarting or rescheduling the job. At the same time, the autonomous hardware recovery service isolates the problematic node, records the reason, and starts diagnostics. That hardware path can continue even after the workload is already moving again. The value of this path is reduced downtime. The workload doesn’t wait for root-cause analysis before recovery begins, and the faulty node is not accidentally handed to the next job. Scenario 2. Transient software fault Not every job failure should trigger hardware remediation. A training worker might exit, hit a recoverable timeout, or lose coordination for a moment even though the underlying node is healthy. In this path, the autonomous job recovery service uses job context to avoid overreaction. If the failure history looks transient, the system retries according to scheduler policy instead of quarantining hardware that doesn’t need repair. While NVIDIA Run:ai may express that recovery differently from how it is handled for Slurm, the intent is the same: return the workload to productive execution without unnecessary hardware remediation. The autonomous hardware recovery service keeps monitoring in the background, but it stays out of the recovery path unless new evidence points to hardware. The value of this path is restraint. The system restores the workload without creating unnecessary operator noise or shrinking the scheduling pool. Scenario 3. Hardware degradation caught early The autonomous hardware recovery service in Mission Control can also detect infrastructure degradation before it becomes a job failure. A node may start showing repeated GPU health warnings, connectivity problems, or other signals that make it risky for future scheduling. In this path, the autonomous hardware recovery service can remove the node from scheduler eligibility proactively and start diagnostics before an AI training job lands there. The autonomous job recovery service may never need to respond, because the faulty capacity is removed from circulation before it disrupts a workload. In this case, Mission Control’s Kubernetes-based scheduler benefits from a cleaner pool of usable resources. It can avoid placing Kubernetes workloads on suspect capacity similarly to how Mission Control has enabled Slurm to keep faulty nodes out of allocations until they are validated again. The value of this path is prevention. The system reduces the chance that a known-risk node becomes tomorrow’s AI training outage. Operational impact The extension of Mission Control’s autonomous recovery engine to NVIDIA Run:ai marks another step towards achieving a self-healing AI factory. This structure changes resiliency from a scheduler-specific incident workflow into a scheduler-aware operating model. For platform teams, Mission Control 2.5 delivers a consistent recovery model across both of their Kubernetes and Slurm estates. Faulty nodes are isolated, annotated or recorded, diagnosed, and returned to service only when they are ready. For AI practitioners, now both Slurm and Kubernetes users can benefit from simpler recovery. Jobs can restart, retry, or requeue according to the scheduler they already use, while the autonomous recovery engine provides the context needed to avoid repeated failures on infrastructure that needs to be serviced. For the business, the benefit is better utilization of valuable accelerator capacity. The system spends less time stalled in ambiguity and more time keeping training workloads productive. Video 1. NVIDIA Mission Control demo of autonomous resilience for Kubernetes-based workloads Get started with Mission Control 2.5 Mission Control 2.5 is available today. Existing NVIDIA Mission Control users can visit the documentation hub to learn more about how to move to the latest version. In addition to extending the autonomous recovery engine to Kubernetes-based workloads, other release highlights include: - New AI-enabled network anomaly detection service (beta only) - Improved user experience for authentication across services using central keycloak - Additional security and Kubernetes hardening updates - Early access for NVIDIA Vera Rubin NVL72 Read the full release notes here or read the solution overview to learn more about how NVIDIA Mission Control simplifies AI factory operations with enterprise-ready software that spans workload orchestration, end-to-end monitoring, and autonomous recovery.
document fetched 2026-10-27 02:54 · canonical https://developer.nvidia.com/blog/how-nvidia-mission-control-2-5-brings-autonomous-recovery-to-kubernetes