How to Read Data into GPU Memory in Python with NVIDIA GPUDirect Storage

retained text (truncated)

GPU applications are processing ever-growing datasets with increasingly powerful accelerators. According to the 2025 Stanford AI Index, compute for notable AI models has been doubling approximately every five months, while the size of datasets used to train LLMs has been doubling approximately every eight months. As computation and data scale together, moving data efficiently from storage becomes a necessity for keeping accelerators supplied.
NVIDIA Magnum IO GPUDirect Storage (GDS) addresses this challenge by enabling a direct data path between supported storage and GPU memory. This avoids unnecessary staging through CPU memory and reduces overall CPU utilization in the system. GDS is supported by a growing ecosystem that includes many major storage vendors.
At the application layer, Python plays a central role in modern AI development. As of August 2025, nearly half of all new AI projects on GitHub were built primarily in Python while the main application interface to GDS is the cuFile C API.
NVIDIA KvikIO, a Python and C++ library for high-performance file I/O, bridges this gap. It provides high-level, idiomatic Python and C++ interfaces, along with additional controls over how I/O requests are performed.
The tutorial in this post demonstrates how to use the NVIDIA KvikIO Python API to configure storage I/O, read data into GPU memory, and confirm that the accelerated path within GDS was activated. It also shows how to inspect the KvikIO and cuFile settings that influence the path each operation takes.
How does NVIDIA KvikIO leverage GDS?
GDS is the broader technology that enables accelerated data movement between supported storage and GPU memory. For file system-based storage, cuFile provides the C library and API through which applications and frameworks access GDS. KvikIO sits above cuFile, providing higher-level Python and C++ interfaces along with its own controls over how I/O requests are executed. Figure 1 shows these layers and their two separate path-selection points. Initialization failures, unsupported configurations, and error-handling paths are omitted for clarity.
The application supplies a file system path and GPU buffer, and KvikIO determines whether to send the request through cuFile or use its POSIX compatibility path, which stages the data through host memory. When cuFile is selected, KvikIO registers the file and issues the I/O using the target GPU buffer.
cuFile then determines the actual transfer path based on system support, the file system, and its configuration. It may use the accelerated GDS path or its CPU-staged compatibility path. Therefore, reaching the cuFile API confirms that KvikIO selected cuFile, but does not by itself confirm that the accelerated GDS path was activated.
The following sections walk through how to configure the system, perform a KvikIO read, and verify the resulting path.
Prerequisites
Before beginning this tutorial, you will need:
- A Linux system with an NVIDIA GPU and GPUDirect Storage installed.
- A GDS-supported storage and file system configuration.
- Permission to create, read, and write files at the storage endpoint, along with access to GDS tools such as gdscheck and gds_stats.
For complete installation and platform requirements, refer to the NVIDIA GPUDirect Storage documentation.
Tutorial system
This tutorial was tested with the following configuration:
- An x86_64 system with an NVIDIA H100 GPU, CUDA 13.3, and cuFile 1.18.
- A local NVMe device formatted with ext4.
- A PIX connection between the selected GPU and NVMe device, meaning the path traverses at most one PCIe bridge.
- IOMMU and ACS disabled to preserve PCIe peer-to-peer transfers.
- The nvidia-fs kernel module is loaded to provide the accelerated path.
You can inspect the GPU-to-storage topology with the following command:
$ nvidia-smi topo -nvme
Step 1: Set up NVIDIA KvikIO
First, create a Python environment and install the CUDA 13 build of KvikIO. CuPy will be installed as a dependency.
$ python3 -m venv kvikio-env
$ source kvikio-env/bin/activate
$ python -m pip install kvikio-cu13
You can verify the installation with the following:
$ python -m pip show kvikio-cu13 cupy-cuda13x
Step 2: Inspect the configuration
Before performing any I/O, inspect the active KvikIO settings and confirm that the system recognizes the GDS components. In the KvikIO defaults API, the compatibility-mode setting is named compat_mode, and its name attribute reports the active mode as AUTO, ON, or OFF. The following commands report the loaded KvikIO and cuFile versions, the active compatibility mode, and the minimum request size KvikIO sends to cuFile:
import kvikio
import kvikio.cufile_driver as cufile_driver
import kvikio.defaults as defaults
kvikio.__version__
cufile_driver.libcufile_version()
defaults.get("compat_mode").name
defaults.get("gds_threshold")
On the tutorial system, KvikIO reports compatibility mode as AUTO and a GDS threshold of 16 KiB. AUTO instructs KvikIO to try cuFile and fall back to its POSIX backend if cuFile is unavailable. Reads smaller than the threshold use the KvikIO POSIX path directly. These settings can also be controlled through KVIKIO_COMPAT_MODE and KVIKIO_GDS_THRESHOLD. For the complete list, refer to the KvikIO runtime settings.
Next, inspect the GDS platform and cuFile configuration:
$ /usr/local/cuda-13.3/gds/tools/gdscheck.py -p 2>&1 | less -R
The output lists the installed GDS components, supported file systems, detected GPUs, and active cuFile configuration. Figure 2 shows how to inspect the KvikIO and GDS configuration used in this tutorial.
KvikIO settings can also be changed in Python. For example, defaults.set accepts a dictionary to change several settings at once, or a name and value to change one setting:
defaults.set({"task_size": 1024**3, "num_threads": 8})
defaults.set("compat_mode", kvikio.CompatMode.OFF)
Step 3: Read a file into GPU memory
Next, create the input file once before running the Python example. The following command writes a 1 GiB file with random data using direct I/O and flushes it to storage:
$ dd if=/dev/urandom \
    of=/mnt/nvme1/kvikio-tutorial/kvikio-example.bin \
    bs=1M count=1024 \
    oflag=direct conv=fsync \
    status=progress
The following program allocates a 1 GiB CuPy array in GPU memory and supplies it to KvikIO as the destination buffer:
from pathlib import Path
import cupy as cp
import kvikio
file_path = Path(
    "/mnt/nvme1/kvikio-tutorial/kvikio-example.bin"
)
# Allocate a GPU buffer large enough to hold the file.
gpu_buffer = cp.empty(
    file_path.stat().st_size,
    dtype=cp.uint8,
)
# Read the file into the GPU buffer.
with kvikio.CuFile(file_path, "r") as file:
    bytes_read = file.read(gpu_buffer)
# Validate that 1 GiB was read.
assert bytes_read == gpu_buffer.nbytes
The context manager opens the file and completes the read into GPU memory. Although the return value confirms that the complete file was read, it does not identify the underlying data path.
Step 4: Determine the data path
One way to validate the data path used by any cuFile-based application is to observe the cuFile runtime per-process statistics. These statistics can be enabled by setting CUFILE_STATS_LEVEL=3 and then inspected with gds_stats while the process remains active.
For this KvikIO example, the environment variable keeps the setting scoped to a single run, although it can also be configured in cufile.json.
$ CUFILE_STATS_LEVEL=3 python3 -i read_with_kvikio.py
Running Python interactively keeps the process alive after the read, allowing gds_stats to inspect its accumulated counters.
Figure 3 demonstrates this workflow. In the per-GPU output, nvfs counts operations representing the path taken through the nvidia-fs kernel module, while posix counts operations handled by the cuFile compatibility path.
The KvikIO default 4 MiB task size divides the 1 GiB request into 256 read operations. The per-GPU statistics report n=256, nvfs=256, and posix=0, showing that all 256 operations used the nvidia-fs path.
Step 5: Compare the available paths
The same read can be repeated with different settings to demonstrate the two control layers. First, require KvikIO to use cuFile:
$ CUFILE_STATS_LEVEL=3 \
    KVIKIO_COMPAT_MODE=off \
    python3 -i read_with_kvikio.py
On the tutorial system, gds_stats reports nvfs operations, confirming that cuFile selected the accelerated path. Next, force cuFile to use its compatibility path:
$ CUFILE_STATS_LEVEL=3 \
    KVIKIO_COMPAT_MODE=off \
    CUFILE_FORCE_COMPAT_MODE=true \
    python3 -i read_with_kvikio.py
KvikIO still calls cuFile, but the operations now appear under posix rather than nvfs. Finally, the KvikIO compatibility mode bypasses cuFile entirely:
$ KVIKIO_COMPAT_MODE=on \
    python3 read_with_kvikio.py
In this case, KvikIO performs the read through its CPU-staged POSIX backend, so cuFile does not record the operation. The next section examines the cuFile settings that control request execution.
Step 6: cuFile configuration for a KvikIO read
After KvikIO selects its cuFile path, cuFile applies another set of controls that determine how the request is executed. These controls are defined in cufile.json and operate independently of KvikIO runtime settings.
Request decomposition and parallel execution
KvikIO and cuFile can both divide a large operation into smaller pieces. KvikIO applies KVIKIO_TASK_SIZE first, with KVIKIO_NTHREADS controlling how those tasks are scheduled. Each task on the cuFile path becomes a separate cuFile request. Once cuFile receives one of those requests, its execution settings determine whether it can be divided again and processed using the cuFile internal thread pool. The following cuFile settings control this behavior:
A 1 GiB read, step by step
The configuration used here has four cuFile worker threads, max_request_parallelism set to four, and min_io_threshold_size_kb set to 8192 (8 MiB). With the KvikIO default 4 MiB task size, a 1 GiB read becomes 256 cuFile requests. Each 4 MiB request is handled as one piece by cuFile. This is the run that produced nvfs=256 in the earlier statistics.
If KVIKIO_TASK_SIZE is set to 1 GiB, KvikIO instead sends one large read request to cuFile. The CuPy destination has not been explicitly registered with cuFile, so cuFile uses its unregistered buffer scheduling path. It limits the request to min (4 worker threads, 4 maximum request parallelism) = 4 pieces.
Dividing 1 GiB four ways gives 256 MiB per piece, already larger than min_io_threshold_size_kb which is 8 MiB. With room in its queue, cuFile assigns three pieces to workers, processes the fourth on the calling thread, and waits for them to finish.
Those 256 MiB pieces are units of scheduled work, not individual storage transfers. Because the destination buffer is not registered for storage DMA, cuFile reads into a registered GPU bounce buffer and copies the data into the destination. Since GDS 1.17, these bounce buffers can be configured as slabs. The nvidia-fs path in this configuration uses slabs no larger than 16 MiB, so each 256 MiB piece is processed through smaller transfers. The 8 MiB scheduling target does not set their transfer size.
When the 8 MiB split target matters
In this configuration, execution.min_io_threshold_size_kb is set to 8192 (8 MiB). That target does not change the 1 GiB split, but it would affect a 16 MiB cuFile request. Four equal pieces would be 4 MiB each, so cuFile schedules two 8 MiB pieces instead. KvikIO must use a task size of at least 16 MiB for cuFile to receive that request intact.
KvikIO in NVIDIA CUDA-X for data science
KvikIO is part of the NVIDIA CUDA-X libraries for data science (formerly RAPIDS). Its development is closely connected to the GDS integration originally built for NVIDIA cuDF GPU DataFrame library, where parallel cuFile reads accelerated Parquet and ORC ingestion. This design was later incorporated into KvikIO, building on the cuDF original GDS data-ingest work.
KvikIO has since been used with GDS i

document fetched 2026-10-24 11:20 · canonical https://developer.nvidia.com/blog/how-to-read-data-into-gpu-memory-in-python-with-nvidia-gpudirect-storage