Build a Validated Kubernetes Runtime for NVIDIA Dynamo

extract

Teams building AI applications need a repeatable path from a model configuration to a working inference endpoint. NVIDIA Dynamo is a distributed inference framework that coordinates model serving across GPU workers. Deploying it on Kubernetes is only a first step. A successful model deployment requires more: GPU drivers, resource allocation, scheduling, networking, and monitoring must all work together. Resolving these dependencies takes time away from testing model behavior, serving performance, and application integration. The NVIDIA DSX AI Factory Platform brings together technologies to help the ecosystem design, build, and operate more efficient AI factories, and Dynamo is one of them. To make management of those infrastructure dependencies easier, the platform contains NVIDIA AI Cluster Runtime (AICR). It packages NVIDIA Dynamo and its supporting infrastructure into validated recipes. Each recipe defines a configuration for a specific environment and workload, including pinned component versions, compatibility constraints, and validation checks. AICR gives platform teams a repeatable way to deploy and check the runtime, providing developers with a consistent foundation for serving models and testing applications. This walkthrough covers the infrastructure that supports Dynamo on Kubernetes, compares two routing options, and shows how to deploy and validate the runtime with AICR. Why the underlying runtime matters for Dynamo Running a model with Dynamo on Kubernetes starts with a DynamoGraphDeployment that specifies how vLLM, SGLang, or NVIDIA TensorRT LLM should run in pods on the cluster. But the graph is only the top of the stack. A production inference cluster also needs a runtime layer below it. It has to answer questions like: - Is the NVIDIA GPU Operator installed with a driver stack that matches the nodes? - Are GPU nodes labeled and discoverable? - Does the cluster expose GPUs through the expected Dynamic Resource Allocation (DRA) driver? - Can the scheduler place GPU workers with the required topology and gang scheduling semantics? - Are node health signals available before workloads land on a degraded node? - Is Prometheus reachable for Dynamo and accelerator metrics? - Is the Gateway API / Endpoint Picker (EPP) path configured when the deployment should be Kubernetes-native at ingress? AICR captures these runtime requirements in the recipe. What AICR ships for Dynamo An AICR Dynamo recipe packages both Dynamo and the cluster runtime components it depends on. AICR ships version 1.4.2 of the dynamo-platform chart, which includes the Dynamo operator, Dynamo runtime images, and resources such as the nvidia.com/v1beta1 DynamoGraphDeployment CustomResourceDefinition (CRD). v1beta1 replaces the former v1alpha1 API version. Table 1 lists the supporting infrastructure components included in the recipe. In addition, the recipe pins component versions and records configuration values, node selectors, tolerations, checksums, and validation expectations to make deployment reproducible. With Dynamo 1.4, frontends, workers, optional Gateway API Inference Extension (GAIE) / Endpoint Picker Plugin (EPP) components, and other Dynamo services discover each other through Kubernetes resources. AICR’s default path therefore no longer uses NATS, and the dynamo-platform chart no longer installs it by default. NATS remains available as an opt-in for deployments that need it. Agentic workloads repeatedly extend long token prefixes, including system prompts, tool definitions, conversation history, and tool results. If a turn reaches a worker without that prefix in GPU memory, the system reprocesses intermediate tokens instead of reusing cached results. Dynamo routes requests to workers that already hold the relevant KV blocks, either through its frontend router or through GAIE/EPP integration at the Gateway API layer. Dynamo can also separate prompt processing (prefill) from token generation (decode), an architecture known as disaggregated serving. On KV-cache misses, all or part of the prompt must be reprocessed by the prefill pool. Decode workers then take over token-by-token generation, which has different GPU, memory, latency, and batching characteristics. The cache miss rate depends on the workload. Long-running agentic sessions behave differently from one-off chat requests. Disaggregated serving enables operators to scale and tune prefill and decode capacity independently for a given request mix. The request plane and KV-cache event plane are separate. In Dynamo 1.4.2, the default AICR Kubernetes path uses TCP for requests and ZMQ for KV-cache events. Each vLLM worker engine emits raw KV-cache events through a local ZMQ publisher, for example, with the following vLLM configuration: --enable-prefix-caching --kv-transfer-config \ {"enable_kv_cache_events": true, "publisher": "zmq", "endpoint": "tcp://*:5557"} Dynamo’s worker runtime then subscribes to that local vLLM event source, normalizes the events, and republishes them onto the Dynamo event plane. Both the native Dynamo Frontend router and the GAIE/EPP path use this Dynamo-managed KV-cache state to make routing decisions. What are the Dynamo routing paths on Kubernetes? Dynamo supports two routing paths on Kubernetes. The first uses the Dynamo Frontend router for worker selection. The second uses a Kubernetes Gateway API implementation for ingress and Dynamo EPP for endpoint selection. AICR ships and validates the Dynamo runtime for either topology. Path 1: Dynamo Frontend router The default path exposes a Dynamo Frontend behind a Kubernetes Service. Running with DYN_ROUTER_MODE=kv, the Frontend tokenizes each request, scores eligible workers using prefix overlap and current load, selects a worker, and dispatches the request through the Dynamo request plane, which uses TCP by default. This path is suitable when an application team owns a model-specific endpoint or an existing edge proxy only needs to forward traffic to the Dynamo Frontend Service. It is the smallest topology for deploying and validating a Dynamo graph and is the most production-proven serving option. It does not require the additional gateway and EPP components used in Path 2. Path 2: Gateway API Inference Extension and Dynamo EPP This path integrates Dynamo with the Gateway API and Gateway API Inference Extension (GAIE). In the AICR example, agentgateway resolves an HTTPRoute to an InferencePool. The Dynamo operator creates that InferencePool from the DynamoGraphDeployment: its selector identifies eligible worker pods, while its endpointPickerRef identifies the Dynamo EPP Service. The gateway invokes the Dynamo EPP through GAIE’s endpoint-selection interface. The EPP tokenizes the request and uses Dynamo’s KV router to select an endpoint. It returns the selection and routing metadata to the gateway implementation, which is the actual HTTP proxy. In this mode, each routable worker pod includes a Dynamo Frontend sidecar that receives the request from the gateway and forwards it to the local worker container. The Frontend runs with --router-mode direct, so it applies the EPP selection without rerouting. This path is suitable when the platform team manages shared Kubernetes ingress and needs to configure model endpoint worker selection, timeouts, retries, and fallback through Gateway API mechanisms. Note that both routing approaches can work behind a Gateway API HTTPRoute. The difference is which component manages the transport to the worker: the Dynamo Frontend or the Gateway itself. Note: This post covers Dynamo’s operator-managed DynamoGraphDeployment integration with GAIE; the experimental standalone vanilla vLLM on-ramp is a separate integration. In Dynamo 1.4, this path uses GAIE’s Go EPP framework with Dynamo’s Rust KV router embedded through CGO and configured through eppConfig. Outlook: Starting with Dynamo 1.5, a native Rust EPP replaces the Go/CGO bridge. It implements GAIE’s Lightweight Endpoint Picker (ext_proc) interface directly, runs the full Dynamo router in-process, and uses DYN_* environment variables in place of the eppConfig plugin and scheduling profiles. Aggregated and disaggregated routing are handled directly from the worker roles registered through Dynamo discovery. The surrounding Kubernetes topology remains unchanged. Removing the Go plugin framework and CGO boundary also reduces per-request EPP overhead. A client request enters through a gateway, such as agentgateway, and matches an HTTPRoute backed by an InferencePool. The Gateway invokes the Dynamo EPP, which performs just the token-aware KV selection and returns the selected endpoint and routing metadata. The Gateway then forwards the request to the selected worker’s Frontend sidecar in direct mode, which sends it to the vLLM worker over the Dynamo request plane. Worker KV-cache state reaches the EPP through the Dynamo event plane, while the Dynamo operator creates the EPP and InferencePool from the DynamoGraphDeployment. Choosing a path The routing path depends on whether the Gateway or the Dynamo Frontend should handle the forwarding of requests to the workers, including timeouts, retries, and fallback management. Table 2 compares the Dynamo Frontend and Gateway API/EPP paths. Both paths use worker-published KV-cache events for precise cache-aware placement. Dynamo can instead be configured to predict KV-cache state from where it sent each request. Prediction cannot see evictions, so the router’s view can drift from what each worker actually holds, and routing quality can degrade, especially under memory pressure. The AICR configuration in this post enables live events, so both paths route using the workers’ reported KV-cache state. How the AICR recipe flow works AICR separates runtime setup from application deployment. First, it captures the existing Kubernetes cluster and generates a matching Dynamo recipe. The runtime bundle is then rendered, deployed, and validated before the application’s DynamoGraphDeployment is installed and its inference endpoints are tested. Step 1: Snapshot the cluster The snapshot captures the cluster state that AICR uses for recipe selection and validation: aicr snapshot \ --namespace aicr-validation \ --output snapshot.yaml When no placement flags are specified, AICR automatically targets nodes labeled nvidia.com/gpu.present=true and tolerates GPU-node taints. The --node-selector and --toleration flags provide more specific placement when required by the cluster. Step 2: Generate the Dynamo inference recipe AICR uses the snapshot to identify the Kubernetes distribution, GPU accelerator, and GPU-node operating system. The command specifies the workload intent and platform: aicr recipe \ --snapshot snapshot.yaml \ --intent inference \ --platform dynamo \ --output recipe.yaml The generated recipe records pinned component versions, environment-specific configuration, compatibility constraints, and validation checks. For the Amazon EKS Dynamo inference recipe for NVIDIA GB200 NVL72 and NVIDIA GB300 NVL72 systems, AICR v1.0.0 selects the following Helm chart versions: The AICR team tested these component versions together on the specified platforms and hardware. The following validation steps verify that the cluster meets the selected recipe’s requirements. Optional offline check Recipe selection already uses the cluster facts in snapshot.yaml, so validating the generated recipe against that same snapshot adds little. Offline validation is useful when testing a recipe against a different captured cluster snapshot without contacting Kubernetes. It evaluates declarative constraints, while behavioral checks are reported as skipped. aicr validate \ --recipe recipe.yaml \ --snapshot candidate-snapshot.yaml \ --no-cluster \ --output dry-run.json Step 3: Render the runtime bundle AICR renders the recipe into deployment-ready Helm values and manifests: aicr bundle \ --recipe recipe.yaml \ --accelerated-node-selector nodeGroup=gpu-worker \ --accelerated-node-toleration dedicated=worker-workload:NoSchedule \ --accelerated-node-toleration dedicated=worker-workload:NoExecute \ --system-node-selector nodeGroup=system-worker \ --system-node-toleration dedicated=system-workload:NoSchedule \ --system-node-toleration dedicated=system-workload:NoExecute \ --set-json agentgateway:allowedSourceRanges='["<your-cidr>"]' \ --deployer helmfile \ --output bundle These placement flags reflect the test cluster used for this walkthrough. The flags can be adapted or omitted for clusters with different labels and taints. If the cluster has no default StorageClass, the command also requires --storage-class. Step 4: Deploy the runtime For a local deployment without an in-cluster GitOps controller, Helmfile provides a deployment option. With Helm and the helm-diff plugin available locally, the following commands inspect the AICR-generated release graph before applying it: helmfile -f bundle/helmfile.yaml diff helmfile -f bundle/helmfile.yaml apply AICR encodes component dependencies in the generated Helmfile bundle. Helmfile processes those dependency tiers in order and applies independent components within a tier concurrently. Rerunning apply reconciles the installed Helm releases with the generated configuration. If Helmfile is not available, the bundle can be regenerated with AICR’s default Helm deployer. Its generated deploy.sh script is a lightweight fallback for an interactive test drive: aicr bundle \ --recipe recipe.yaml \ [... all the flags setting node selectors and tolerations ...] \ --set-json agentgateway:allowedSourceRanges='["<your-cidr>"]' \ --deployer helm \ --output bundle-helm chmod +x bundle-helm/deploy.sh ./bundle-helm/deploy.sh For production, AICR also renders GitOps bundles with --deployer argocd, --deployer argocd-helm, and --deployer flux. Step 5: Validate the installed runtime After installation, deployment and conformance validation check the live cluster before the application workload is installed. Because no snapshot is supplied here, AICR captures the current cluster state before running the validator Jobs. aicr validate \ --recipe recipe.yaml \ --node-selector nodeGroup=gpu-worker \ --toleration dedicated=worker-workload:NoSchedule \ --toleration dedicated=worker-workload:NoExecute \ --phase deployment \ --phase conformance \ --output report.json The application DynamoGraphDeployment is installed only after these checks pass. Performance validation runs separately because it deploys a temporary serving workload and consumes the GPUs under test. What does AICR validation check? AICR validates the installed runtime against the checks and constraints encoded in the selected recipe. Checks vary by recipe. For a Dynamo inference recipe, deployment and conformance validation typically cover the following: These checks verify that the installed runtime meets the selected recipe’s requirements before an application is deployed. AICR combines a versioned runtime configuration with checks that verify whether the cluster meets the selected recipe’s requirements. Get Started NVIDIA AI Cluster Runtime (AICR) and NVIDIA Dynamo are open-source components of the NVIDIA DSX AI Factory Platform. Teams deploying Dynamo on Kubernetes can use AICR’s recipes and deployment examples to configure, deploy, and validate the infrastructure supporting their inference workload. - Additional projects and community resources are available through the NVIDIA DSX AI Factory GitHub. - Discussion is available in the NVIDIA DSX AI Factory Slack community. The NVIDIA DSX OS AI Inference and Contributor Workshop provides hands-on experience with AICR during KubeCon + CloudNativeCon North America. Registration is required, as seating is limited. - The NVIDIA booth #170 at KubeCon features Kubernetes deep dives and live demos.