NVIDIA Open-Sources Run:ai GPU Orchestration Platform

extract

NVIDIA is making the full NVIDIA Run:ai platform available as open source software, giving customers, developers, and ecosystem partners access to production-grade AI workload and GPU orchestration capabilities. This release expands the NVIDIA DSX AI Factory Platform by establishing NVIDIA Run:ai as an open foundation that enables the broader ecosystem to bring differentiated solutions to market faster in response to evolving customer needs. Over the past several years, NVIDIA Run:ai has helped organizations run AI workloads on Kubernetes at scale by dynamically allocating GPU resources and managing workloads across the full lifecycle, from data processing and training to inference. By open sourcing NVIDIA Run:ai platform capabilities as part of NVIDIA DSX AI Factory, it is now possible for every NVIDIA DSX ecosystem partner and AI cloud to increase workload throughput, improve GPU utilization, and operate shared AI infrastructure more efficiently and effectively. This post details the open source release, which includes the complete NVIDIA Run:ai platform and four components that can also be adopted independently: KAI Scheduler, GPU Fractioning Operator, KAI Resource Management, Workload Map. Together with additional platform code, these components form the integrated NVIDIA Run:ai platform. Platform teams and ecosystem partners can adopt NVIDIA Run:ai as an integrated whole, build on it for their own products and services, or integrate individual components based on their requirements. How to adopt open source NVIDIA Run:ai NVIDIA Run:ai can be adopted directly as open-source software or through NVIDIA DSX ecosystem partner offerings. Deploy the open source platform directly The complete source code for NVIDIA Run:ai is available in the NVIDIA Run:ai repository within the NVIDIA DSX AI Factory GitHub organization. Teams can inspect the code, contribute to its development, build on top of the platform or adopt its modular components independently. The complete open source NVIDIA Run:ai platform is also available on NVIDIA NGC as a Community Product, with a Helm chart for installing the integrated stack. Note: The NVIDIA Run:ai source code is governed by its applicable open source licenses. Downloading the Community Product from NGC requires acceptance of the applicable NVIDIA Software License Agreement and Product-Specific Terms for NVIDIA AI Products. No paid NVIDIA Run:ai subscription or NVIDIA support entitlement is required for the Community Product, and NVIDIA enterprise support is not included. Adopt open source NVIDIA Run:ai through NVIDIA DSX ecosystem partners Open source NVIDIA Run:ai gives NVIDIA DSX ecosystem partners direct access to AI workload and GPU orchestration innovations they can use in their own NVIDIA DSX AI factory solutions. By adopting the complete platform or integrating individual components, partners can bring those innovations to market faster and respond more quickly to evolving customer and operational requirements. - Crusoe integrates NVIDIA Run:ai with Crusoe Managed Kubernetes, combining GPU workload orchestration with managed Kubernetes and GPU infrastructure. - Mirantis extends k0rdent AI with NVIDIA Run:ai to deliver a lifecycle-managed, Metal-to-Model AI factory stack that automates infrastructure provisioning and maximizes GPU utilization. - Nebius offers NVIDIA Run:ai as a one-click application deployment in Nebius AI Cloud, giving customers a streamlined way to deploy and operate the complete platform on Nebius Managed Service for Kubernetes. - OpenNebula Systems integrates NVIDIA Run:ai with the OpenNebula AI Cloud Platform through OneKS, combining GPU workload orchestration with infrastructure provisioning and lifecycle management for open and sovereign AI clouds. - Rafay is expanding its NVIDIA Run:ai integration to deliver it as a fully managed service. Rafay automates the entire stack, from GPU infrastructure and clusters to the NVIDIA Run:ai control plane and tenants, so AI cloud providers and enterprises can offer it through their own self-service catalogs. - Saturn Cloud integrates NVIDIA Run:ai to help AI cloud operators launch white-label AI inference services through its Token Factory platform, combining GPU orchestration with dedicated and multi-tenant model serving, per-token metering, and automated billing. - Spectro Cloud integrates NVIDIA Run:ai into PaletteAI, enabling customers to deploy it through PaletteAI Studio and manage it with the same lifecycle automation used across their AI infrastructure. - vCluster provides NVIDIA Run:ai through pre-validated deployment templates that stand up a complete environment per tenant, allowing operators to strengthen tenant boundaries while NVIDIA Run:ai continues to schedule across one GPU fleet. For NVIDIA Enterprise AI Factory deployments, NVIDIA Run:ai continues to be available through NVIDIA-supported commercial offerings including as a standalone product, through NVIDIA Mission Control and NVIDIA AI Enterprise, and select cloud marketplaces. Availability, licensing, and support terms vary by offering. For teams that already have a platform and need to fill a specific capability gap, the following sections describe when and how to use each modular component independently. Modular open source components of the NVIDIA Run:ai platform This section details the four components included in the open source release: KAI Scheduler, GPU Fractioning Operator, KAI Resource Management, and Workload Map. With additional platform code, these components form the integrated NVIDIA Run:ai platform. You can adopt NVIDIA Run:ai as an integrated whole or integrate individual components based on your unique requirements. KAI Scheduler: The scheduling engine behind NVIDIA Run:ai KAI Scheduler is the Kubernetes-native scheduler that powers the NVIDIA Run:ai platform. NVIDIA released KAI Scheduler as open source software under the Apache 2.0 license and contributed it to CNCF as a Sandbox project. KAI remains the upstream engine behind NVIDIA Run:ai, and all scheduler development happens in the community repository, so every improvement flows through the same codebase. NVIDIA continues to support the project’s maintainers and work with the broader ecosystem to advance AI workload scheduling at scale. KAI Scheduler is purpose-built for GPU workloads at scale, coordinating: - Gang scheduling with prescheduling in-memory simulation, so multipod jobs are placed all-or-nothing and aren’t evicted needlessly - Fair-share (including time-based fair-share) and quota allocation that keeps resource shares balanced across teams - Topology awareness and bin-packing at the scheduling-decision level - GPU sharing, which allows multiple workloads to share single or multiple GPUs (MIG, MPS, time sharing, HaMi, CUDA 13.4 MPS fractioning) - Dynamic Resource Allocation (DRA) support To learn more about key features, visit the kai-scheduler/KAI-Scheduler GitHub repo. Use KAI Scheduler for diverse AI workloads that require fair access to cluster resources and topology-aware placement. It can be deployed in an existing cluster alongside the current software stack without requiring the complete NVIDIA Run:ai platform. Follow the project, ask questions in the #kai-scheduler channel on CNCF Slack, and contribute on the kai-scheduler/KAI-Scheduler GitHub repo. GPU Fractioning: An operator for GPU fractions on Kubernetes The NVIDIA Run:ai platform has long supported fractional GPU allocation, enabling multiple workloads to share a GPU with hard memory isolation. Fractions can be requested on demand and set up on the fly, with no per-node setup from an administrator in advance. It is one of the most widely used capabilities among ML teams running workloads that do not require the full memory of a GPU, including interactive notebooks and inference for smaller models. NVIDIA Run:ai pioneered that experience with a proprietary implementation. GPU Fractioning Operator brings CUDA-native GPU fractioning to Kubernetes and the proprietary implementation has been retired in favor of the native capability available in CUDA 13.4. The CUDA driver enforces a hard limit on GPU memory for each workload, and each workload sees only the GPU memory allocated to its fraction. What is still needed is a clean way to consume it on Kubernetes. This is what GPU Fractioning Operator provides: an operator that enables CUDA GPU fractions on a cluster, along with a metrics exporter for per-pod visibility into fractions in use. It is a Kubernetes integration layer that allows workloads to use CUDA-native fractioning, and it is scheduler-agnostic. Use the GPU Fractioning Operator for workloads that can share underutilized GPU capacity, including notebooks, development sessions, and smaller models. GPU Fractioning Operator integrates with KAI Scheduler to translate available GPU capacity into an actual placement decision, while remaining compatible with other schedulers. GPU Fractioning Operator brings CUDA 13.4-native GPU fractioning to Kubernetes and integrates with KAI Scheduler. To get started, refer to the GPU Fractioning Operator documentation. KAI Resource Management: The organizational layer While the scheduler decides how work is placed, KAI Resource Management governs who gets what: - Projects and departments: The organizational structure, or “org layer,” that maps resources to the teams and initiatives that consume them. - Resource management with an organizational hierarchy: The ability to allocate, guarantee, and share GPU capacity across that org structure, with fair-share and quota semantics. - Node pools: Grouping and targeting of heterogeneous hardware so workloads land on the right resources. Together, these let platform teams move beyond raw Kubernetes scheduling and manage GPUs the way organizations actually operate: by team, by department, by initiative, and by resource guarantee. Use KAI Resource Management for heterogeneous clusters shared by multiple teams that require defined shares of capacity. For example, a platform team could guarantee 50% of the cluster to research while preventing any one team from monopolizing the remaining capacity. If you are already tracking quota entitlements in a spreadsheet or hand-maintaining namespaces per team, this is the layer that replaces it. It can be combined with KAI Scheduler in your clusters. Read more about how these concepts map to real deployments in the KAI Resource Management documentation. Workload Map: A workload-management contract for Kubernetes Workload Map is the contract NVIDIA Run:ai uses to support any workload type. KAI Scheduler uses it for scheduling and gang semantics; the rest of the platform uses it for workload discovery, monitoring and lifecycle management. In Kubernetes, an AI workload is almost never a bare Deployment or Job. It is a PyTorchJob, a RayCluster, a JobSet, a KServe InferenceService, a Dynamo graph deployment, or a company’s own custom resource. Each type has its own internal structure, components with different roles (driver and workers, head and workers, prefill and decode), its own scaling and gang semantics, and its own way of reporting status. Workload Map provides a consistent way to treat these workloads as first-class objects throughout their lifecycle: - Describing pods, components, and roles - Submitting and configuring - Scheduling with gang and topology awareness - Observing status and health consistently - Applying policy at admission or runtime This workload-management layer sits between Kubernetes primitives and the schedulers, dashboards, SRE and troubleshooting tools, cost tools, backup tools, and ML platforms that serve users. Instead of each tool building a private, partial integration for every CRD, Workload Map defines each workload type once so any platform or tool can schedule, observe, act on, and govern it consistently without hardcoding each CRD. Use Workload Map for platforms and tools that must understand and manage multiple Kubernetes workload types without hardcoding each CRD. It ships as a Go library that works offline on unstructured objects, so you can adopt it without adding anything to your cluster. It is deliberately narrow: Workload Map describes workload structure and does not schedule, queue, autoscale, or propagate across clusters. Learn more in the dsx-ai-factory/workload-map GitHub repo and check out the adopter projects. Related open source platform software The following open source projects address complementary AI infrastructure challenges. They can be adopted independently or incorporated into broader solutions that include the complete NVIDIA Run:ai platform or selected components. Model Streamer: Reducing cold-start latency Cold-starting an inference workload on Kubernetes can take several minutes, during which time GPUs are allocated but idle for a single step: loading model weights from storage onto the GPU. Model Streamer reads weights concurrently and streams them directly into GPU memory, cutting load time for large models. Since its release, Model Streamer has been adopted across the inference stack, including by inference engines, cloud providers and storage platform vendors. Inference engines Model Streamer is integrated upstream in vLLM and SGLang, where it can be enabled with the --load-format runai_streamer flag, as well as in ModelExpress. Support covers local filesystems, S3-compatible stores, and network file shares alongside several cloud backends detailed below. Clouds Major cloud providers now document Model Streamer in their own guidance. - AWS: Model Streamer ships in the vLLM Deep Learning Container and is the recommended fast path in the official EKS documentation for running AI/ML inference, with tuned concurrency and chunk sizing for S3. - Google Cloud: Native Cloud Storage support built in collaboration with NVIDIA, including GKE documentation for streaming straight from a bucket into accelerator memory. - Microsoft Azure: Azure Blob Storage integration, jointly developed by Microsoft and NVIDIA Run:ai to provide first-class model streaming for Azure customers. Published benchmarks on eight NVIDIA H100 VMs shows 3x-6x faster loading than the default loader and a steady 80 Gbps, compared with approximately 40 Gbps for the default path. Storage platforms Several AI data infrastructure partners have independently benchmarked Model Streamer and published their results: - WEKA loaded into GPU memory in just 3.5 seconds, roughly 40% faster than loading from local NVMe—enabling organizations to build petabyte-scale shared model repositories that accelerate model readiness without the time, effort, and capacity required to maintain duplicate model copies across the inference environment. - VAST Data measured measured 1.7 seconds, a 60% improvement over loading from local NVMe - DDN streamed the 65.5 GB Qwen3-32B across 8 GPUs in under four seconds, and translated it into cost: saving up to 10% of daily GPU runtime compared to using local storage. Use Model Streamer when starting new replicas and loading model weights is the primary bottleneck. Grove: One API, any inference architecture Emerging workload patterns—including agentic pipelines, disaggregated serving and mixture-of-experts training—consist of multiple connected components that must scale together. Grove provides a Kubernetes API for orchestrating these workloads, from a single pod to deployments spanning tens of thousands of GPUs. Grove provides topology-aware autoscaling and hierarchical gang scheduling, allowing components such as prefill and decode to start as all-or-nothing units. On NVIDIA GB200 NVL72, it also manages ComputeDomain resources automatically, enabling multinode NVLink performance through a single annotation instead of hand-authored YAML. Grove is a modular part of NVIDIA Dynamo that can be used independently, and it already powers training and inference pipelines at scale across a range of companies. Recent work includes support for llm-d and Volcano as a scheduler backend, with Kueue support in progress. Use Grove for graph-based workloads with distinct roles and coupled scaling requirements. Get started with open source NVIDIA Run:ai NVIDIA Run:ai is available now as open source software in the NVIDIA DSX AI Factory GitHub organization. To learn more and get started, check out these resources: - NVIDIA Run:ai full platform - KAI Scheduler scheduling engine - KAI Resource Management for projects, departments, quotas, and node pools - GPU Fractioning Operator for CUDA GPU fractions on Kubernetes, with per-pod metrics - Workload Map workload-management contract Related open source projects include: - Model Streamer for fast model weight loading - Grove for multicomponent inference orchestration We’re excited to bring this work to the community and to build the next generation of AI workload and GPU orchestration in the open. We welcome your contributions, issues, and feedback.