loading
loading
Many AI workloads need only a fraction of a GPU. Giving each one an entire device can leave capacity idle; sharing GPUs lets platform teams serve more workloads with the infrastructure they already have.
NVIDIA GPUs already support several approaches to sharing. Multi-instance GPU (MIG) splits a GPU into hardware-isolated instances that are transparent to the application, time-slicing lets several processes take turns without memory enforcement, and Multi-Process Service (MPS) lets them run concurrently in separate address spaces (sm-sharing).
CUDA 13.4 now enforces a memory boundary that is transparent to the application with a new time-sharing mode, alongside the compute sharing mode (sm-sharing) that already existed in MPS.
Here is what the CUDA release brings:
- An exact memory amount: A workload asks for a memory size and CUDA holds it there. An allocation above that size fails, just as it would on a smaller GPU.
- A range: A workload gets a guaranteed minimum plus a maximum it can grow into when memory is free.
- A compute mode: Each workload can choose to use its compute in an sm-sharing mode rather than using the default time-sharing mode.
- A transparent memory limit: Each workload sees only a fraction of the GPU memory.
The missing piece: Getting there from Kubernetes
CUDA provides the memory enforcement capability, but a Kubernetes cluster can’t use it on its own. On every GPU node, something has to:
- Run and supervise the daemons that these features depend on
- Apply each Pod container’s memory limits and compute mode before the process starts
- Report what each fraction actually uses, rather than what the whole GPU uses
This work belongs on the node, independent of whichever scheduler placed the workload.
GPU fractioning: An open-source Kubernetes operator
A new open-source Kubernetes operator, gpu-fractioning, is hosted under the KAI Scheduler organization on GitHub. It runs next to the NVIDIA GPU Operator, which provides the driver and the container runtime setup.
GPU Fractioning Operator handles three jobs:
- It sets up and runs MPS on every GPU node.
- It applies each workload’s memory limits and compute mode.
- It measures memory and compute use per fraction rather than per GPU.
How schedulers pass GPU allocations to the operator
The developers write the resource and mode annotations in the pod YAML. Then the scheduler selects the GPUs for a workload and records its memory allocation and compute-sharing mode in per-container Pod annotations. GPU Fractioning Operator reads those annotations and configures the assigned GPUs to apply the workload’s memory limits and compute mode:
Any Kubernetes scheduler that can write these annotations can enforce GPU fractions with CUDA. The scheduler still has its own job: choosing a GPU with enough free memory and never mixing compute modes on one GPU.
Using GPU Fractioning with KAI Scheduler
KAI Scheduler is the first scheduler integrated with gpu-fractioning. Starting with v0.18.0, KAI Scheduler has a GPU-sharing mode called NvFractions. KAI Scheduler picks which slice of which GPU each workload gets and writes the annotations (see Table 1, above), and gpu-fractioning enforces them on the node.
KAI Scheduler’s other sharing modes stay available, and administrators choose one per cluster. It can be enabled when KAI is installed:
helm upgrade -i kai-scheduler oci://ghcr.io/kai-scheduler/kai-scheduler/kai-scheduler \
-n kai-scheduler --create-namespace \
--set global.nvFractions.set=true
This also installs gpu-fractioning as a dependency. From then on, you describe a fractional workload with Pod annotations.
There are four important annotations that answer four questions:
1. How much memory is needed
2. How far the workload may grow
3. How compute is shared
4. How many GPUs it spans
How much memory: The request
There are two ways to request memory:
1. As an absolute size: Use the per-container NVIDIA annotation and standard Kubernetes units:
The container name (app) is part of the key, so it always targets the right container.
2. As a portion of a GPU by using KAI Scheduler’s existing annotation:
gpu-fraction: "0.5"
How far it may grow: The limit
This ability is new in KAI Scheduler. A workload can declare a guaranteed minimum (the request) and a maximum it may burst into when memory is free. Which limit annotation you use depends on how you wrote the request:
The limit is optional. If it is omitted, the limit equals the request and the workload is capped at exactly what it asked for. A portion limit must be greater than the gpu-fraction and less than 1.
How compute is shared: Time sharing or sm-sharing
Compute can now be divided. Call n the number of workloads sharing a GPU: either way each gets roughly 1/n of the compute, and what differs is whether they take turns or run together.
- Time-slicing is serial: Workloads take turns on the whole GPU, and when every one of them uses its full slice each gets about 1/n of the time. Time left unused by one workload is shared equally between the others. This is the default. Each workload runs in its own error domain, i.e. full error isolation.
- SM-sharing is parallel: The GPU’s streaming multiprocessors (SMs) are split between workloads and all run at once, continuously. Nothing is carved out in advance. The GPU hardware allocates and releases SMs as workloads need them. When every workload is busy the split settles at roughly 1/n; when one goes quiet its SMs go to the others, and a single active workload can use them all. In this mode, workloads share a single error domain.
Compute mode is now explicit and can be set per container:
nvidia.com/container.app.gpu-compute.mode: "sm-sharing" # or "time-slicing"
A shared GPU runs one compute mode at a time, so KAI Scheduler never places a time-slicing workload and an sm-sharing workload on the same GPU. KAI Scheduler tracks this per GPU, not per node, so one node can host both modes on different GPUs.
How many GPUs are utilized
To get fractional capacity on several devices, add:
gpu-fraction-num-devices: "4"
The request applies to each device. A 40Gi request with four devices gets 40Gi on each of four GPUs.
Putting it together: Examples of GPU fractioning
An absolute size with room to burst on four GPUs with SM-sharing:
apiVersion: v1
kind: Pod
metadata:
name: inference
labels:
kai.scheduler/queue: default-queue
annotations:
nvidia.com/container.app.gpu-memory.request: "40Gi"
nvidia.com/container.app.gpu-memory.limit: "60Gi" # optional
nvidia.com/container.app.gpu-compute.mode: "sm-sharing" # optional, default time-slicing
gpu-fraction-num-devices: "4" # optional, default 1
spec:
schedulerName: kai-scheduler
containers:
- name: app
image: nvcr.io/nvidia/cuda:13.0.2-base-ubuntu24.04
This is an inference server split across four GPUs that needs a guaranteed 40Gi on each GPU, can use up to 60Gi when neighbors are idle, and runs alongside other workloads instead of waiting its turn. SM-sharing puts the workloads in one error domain, so use it for workloads you trust to run together.
Half a GPU that may grow to 80% on two GPUs with time slicing (default):
metadata:
annotations:
gpu-fraction: "0.5"
gpu-fraction.limit: "0.8" # optional
nvidia.com/container.app.gpu-compute.mode: "time-slicing" # optional
gpu-fraction-num-devices: "2" # optional
spec:
schedulerName: kai-scheduler
containers:
- name: app
This is an example for a development or batch job, such as notebooks or small fine-tuning runs, that need a guaranteed half of each of two GPUs and can grow to 80% when memory is free. Time-slicing gives each workload its own error domain, so a crash in one job doesn’t affect its neighbors. Running nvidia-smi in the container should show two GPUs, each with this workload’s fractional memory share.
Getting started
KAI Scheduler and the GPU Fractioning Operator are both on GitHub under the KAI Scheduler organization, with installation and configuration instructions for all three sharing mechanisms (See admin and user guides). Join the Slack channel or bi-weekly community meetings to ask your questions and contribute.
For more information about CUDA 13.4, refer to the announcement.
You can also get involved during KubeCon + CloudNativeCon NA 2026 in Salt Lake City, Utah, Nov. 9-12.document fetched 2026-10-28 02:51 · canonical https://developer.nvidia.com/blog/memory-enforced-gpu-sharing-in-kubernetes-with-nvidia-cuda-13-4