Manage Kubernetes Node Fleets with NodeWright

extract

Kubernetes manages what runs on your nodes. Managing the nodes themselves is the challenge: kernel settings, system packages, storage layouts, security agents, and the host-level tuning that GPU workloads depend on. Many teams manage this with Ansible playbooks, custom scripts, and manual runbooks. That works until a new cluster comes up in a different region, a kernel upgrade breaks RDMA, or a CVE has to be remediated across the whole fleet this week. GPU infrastructure makes the challenge harder. You cannot simply discard a GPU node and spin up a fresh one. Hardware is scarce, replacements can take hours, and long-running training jobs cannot simply be rescheduled. So the operational question is not how to change a node; it is how to change all of them without killing the training run. Today, that answer is usually a spreadsheet, a maintenance window, and an engineer watching a terminal at 3 a.m. The NVIDIA DSX platform helps you design and operate AI factories, where an entire facility is treated as a single system rather than a collection of machines. Host configuration follows the same logic: the unit of change is the fleet, not the node. NVIDIA DSX OS is the open-source software layer, and NodeWright is the project within it that configures and updates the underlying host operating systems. From scripts to declarative node management DSX OS addresses this challenge through a modular portfolio of open-source projects spanning the AI-ready foundation, resource and workload orchestration, and production AI services. The AI-Ready Foundation defines, configures, validates, and assures accelerated Kubernetes infrastructure. NVIDIA GPU Operator, NVIDIA Network Operator, Topograph, DRA Driver for NVIDIA GPUs, NodeWright, NVIDIA Cluster Readiness Engine (NVCRE), and NVSentinel provide the supporting infrastructure capabilities. NodeWright declaratively configures and safely updates Kubernetes node operating systems without disrupting workloads. It is an open-source, Kubernetes-native package manager for modifying and maintaining host infrastructure at scale. Think of NodeWright as apt or yum for an entire cluster: aware of workloads, aware of disruption budgets, and able to roll changes out progressively across a fleet. NodeWright has run in production at NVIDIA as Skyhook and is available as open source. This post officially introduces the NodeWright name. Why Kubernetes needs its own package manager Excellent tools already exist for configuring machines. Ansible and Puppet have done it for years. They were designed for a world where machines are managed individually, not as part of a cluster that is actively running sensitive workloads. When you need to update a kernel parameter across 200 GPU nodes, the hard part is not running the script. It is doing so without disrupting the training jobs on those nodes. Traditional configuration management is not aware of Kubernetes. It will not cordon a node before making changes, wait for a critical pod to finish, or drain workloads before rebooting. It will not track success or failure inside the cluster, where the rest of your observability already lives. NodeWright closes that gap. It manages the full lifecycle of host-level changes, from installation through configuration, upgrade, and uninstallation. It does so while respecting the Kubernetes primitives your workloads already depend on: PodDisruptionBudgets, node selectors, taints, and tolerations. NodeWright packages are defined as Custom Resources, so they deploy the way everything else in your cluster does: through kubectl, Helm, Argo CD, Flux, or whatever GitOps tooling you already run. How NodeWright works NodeWright has three main components: the operator, custom resources, and packages. The operator is a Kubernetes controller that watches for NodeWright custom resources and manages the lifecycle of changes across your nodes. Packages are container images that carry the actual modifications: scripts, configurations, and binaries. Packages also include verification scripts that surface failures and stop the rollout when a modification is incorrect. apiVersion: nodewright.nvidia.com/v1alpha1 kind: NodeWright metadata: name: gpu-node-tuning spec: nodeSelectors: matchLabels: nodepool: gpu interruptionBudget: percent: 33 podNonInterruptLabels: matchLabels: workload: long-running-training packages: nvidia-tuned: version: 0.9.0 image: ghcr.io/nvidia/nodewright-packages/nvidia-tuned When you apply a NodeWright custom resource, the operator orchestrates a careful sequence on each targeted node. Figure 1 shows the full sequence and how a protected workload pauses it. - Cordon. Mark the node as unschedulable so no new workloads land on it. - Wait. Let critical workloads finish gracefully. You declare which pods must never be interrupted by label. - Drain. Evict remaining pods, honoring PodDisruptionBudgets by default, with configurable drain behavior when the defaults are insufficient. - Apply and configure. Run the package: set kernel parameters, install agents, and configure system services. - Interrupt. If the change requires it, restart a service or reboot the node. - Uncordon. Return the node to the cluster, ready for workloads. This sequence is the difference between losing half a training run to a kernel update and landing the same update across a fleet without workload disruption. NodeWright packages can perform many host-level operations that would normally require root access without recycling nodes. They can set sysctl and GRUB parameters, configure crash dump collection, create logical volumes, install security agents, remediate CVEs, and perform other system-level configuration tasks. NodeWright tracks the state and semantic version of every package on every node, allowing it to distinguish between a fresh installation, an upgrade, and a downgrade. Packages can also declare dependencies, which NodeWright uses to determine the correct execution order. Validation is built into the package lifecycle. Apply, configuration, upgrade, uninstall, and post-interrupt work can each be paired with a check that verifies the expected node state and surfaces failures through Kubernetes. For example, a CVE remediation package can detect whether a vulnerable kernel module remains loaded and mark the package as failed, giving operators immediate visibility into affected nodes rather than leaving them to discover the problem later through workload failures. This same model extends to node readiness as clusters scale. Newly provisioned nodes can otherwise become schedulable before required configuration and tuning have been applied. NodeWright can require new nodes to join the cluster with a Kubernetes taint, complete the required package operations and validation checks, and remove the taint only after the node passes. This creates a controlled path from provisioned to configured to validated to schedulable, ensuring that new capacity does not accept production workloads until it is genuinely ready. Safe rollouts at scale Pushing a change to one node is straightforward. Pushing it to a thousand GPU nodes running production training jobs is a different problem. The NodeWright DeploymentPolicy resource provides progressive rollout strategies that control how changes propagate across a fleet. You define compartments, which are named groups of nodes selected by labels, each with its own disruption budget and rollout strategy. Figure 2 shows the three available strategies. - Fixed. Constant batch size. Update five nodes at a time, every time. - Linear. Increase the batch by a fixed delta. Start with 1, then 2, then 3, building confidence as you go. - Exponential. Multiply the batch by a growth factor. Start with 1, then 2, then 4, then 8. Fast once you trust it. Each strategy includes a batch threshold, the minimum success percentage required before advancing to the next batch. An optional failure threshold stops that compartment if too many consecutive batches fail. You set the risk tolerance. NodeWright enforces it. This means you can start a fleet-wide kernel update with a single canary node, verify that it is healthy, and let the rollout accelerate automatically. If something goes wrong, the update stops rather than cascading. NodeWright reports errors in its status, marks failed Jobs, and adds a label and condition to each affected node. You can quickly identify and triage the cause through the Kubernetes API. NodeWright packages and NVIDIA AI Cluster Runtime NodeWright functions as a versatile package management platform. Because packages execute operations that require root-level privileges, host modification relies directly on native Kubernetes primitives: fine-grained RBAC controls user permissions, admission controllers validate specifications, and integrated validation checks ensure state consistency at every step. The public package repository provides modular foundational components for running shell commands, managing bind mounts, and establishing kernel crash dump collectors. NVIDIA also publishes packages based on operational knowledge that NVIDIA teams have developed while running GPU clusters at scale. Tuning packages use an intent-based model. Rather than naming a profile, you declare your accelerator and what you are doing with it, such as NVIDIA Blackwell GPUs and multi-node training. The package automatically assembles the right profile: kernel parameters, power management, and system settings. Coverage spans NVIDIA Hopper and Blackwell GPUs, with a generic baseline profile for any NVIDIA GPU and variants for different cloud environments. A dedicated package covers Google Kubernetes Engine (GKE) nodes running Container-Optimized OS, where the usual tuning stack is not available. Node setup packages automate bootstrap steps for specific cloud and accelerator combinations, handling kernel version management and Elastic Fabric Adapter (EFA) driver installation for Amazon Elastic Kubernetes Service (Amazon EKS) clusters with NVIDIA Hopper or Blackwell GPUs. These packages are part of a broader effort. NodeWright integrates with NVIDIA AI Cluster Runtime (AICR). AICR captures known-good combinations of drivers, operators, kernels, and system configurations and publishes them as version-locked recipes. Its component catalog pins both the NodeWright operator and the NodeWright customizations that carry environment-specific tuning, then renders them into deployment-ready bundles for Helm, Argo CD, Flux, or Helmfile. NodeWright applies the host-level parts of those recipes to running nodes. Two companion NVIDIA open-source projects—NVCRE and NVSentinel—complete the ecosystem. NVCRE handles pre-workload validation to verify that accelerated infrastructure is production-ready, whereas NVSentinel monitors for runtime faults and facilitates cordon, drain, and remediation operations. Together with NodeWright, these tools support provisioning, maintenance, and self-healing for GPU-accelerated Kubernetes clusters. To be clear about scope: NodeWright does not replace the NVIDIA GPU Operator or NVIDIA Network Operator. It manages the host OS layer underneath them. Get started NodeWright installs with Helm into any Kubernetes cluster. The chart is distributed as an OCI artifact, so there is no repository to add: helm install nodewright oci://ghcr.io/nvidia/nodewright/charts/nodewright \ --version <chart-version> \ --namespace nodewright \ --create-namespace From there, define a NodeWright Custom Resource with the packages you want applied, target your nodes by label, and the operator handles the rest. - NodeWright repository: source, issues, and discussions - Packages repository: NVIDIA and community packages - NodeWright documentation: architecture, CLI reference, and deployment policies - NVIDIA AI Cluster Runtime: the broader validated configuration system Get involved NodeWright is licensed under Apache 2.0 and part of DSX OS, a modular portfolio of open-source projects spanning the AI-ready foundation, resource and workload orchestration, and production AI services. Adopt one project, integrate several, or compose them into a platform. The value lies in open interfaces, independent adoption, and a coherent lifecycle, not in a monolithic stack. Operating this stack at customer scale reveals failure modes early, and NVIDIA teams share those observations with the community. The package repository is where that happens for NodeWright. The NodeWright team is especially interested in packages for hardware and cloud combinations that the catalog does not yet cover and tuning profiles from operators running configurations the team has not yet characterized. Kubernetes transformed how teams manage workloads. The underlying nodes—especially GPU nodes running demanding AI workloads—still rely on scripts and manual runbooks. NodeWright brings the same declarative, automated, safe approach down to the host layer. Adopt what you need. Help shape what comes next. Project references: NodeWright repository and NodeWright packages repository.