# Fast Model Swapping RAM <-> VRAM (<2s) on a Single 24GB GPU for Multi-Agent Workflows: Has anyone solved the process/CUDA re-init bottleneck? — retained record

> Retained reddit record from reddit:LocalLLaMA.

Canonical: https://highsignal.app/data/records/c84bac3f061e8f4e

[← sources](https://highsignal.app/data)


source: [reddit](https://highsignal.app/data/reddit) (reddit:LocalLLaMA)  published 2026-10-31 10:27 ingested 2026-11-01 02:40

[original source ↗](https://www.reddit.com/r/LocalLLaMA/comments/1xk6qbw/fast_model_swapping_ram_vram_2s_on_a_single_24gb/)

## retained text



```
Hi everyone,

I am building a sequential multi-agent local pipeline (Architect -> Coder -> Debugger) on a single GPU workstation, and I am hitting a frustrating software bottleneck. I'd love to know if anyone in the community has solved this or found a working architecture/fork for near-instant model swapping.

Hardware Setup

GPU: 1x NVIDIA RTX 3090 (24 GB VRAM) on PCIe 3.0 x16 (\~12-13 GB/s real throughput).

System RAM: 48 GB DDR4 (\~38 GB free when idle).

Storage: Fast PCIe 4.0 NVMe SSD.

OS: Windows 11.

The Desired Architecture

Because 24 GB VRAM cannot fit two \~18 GB models simultaneously, I run them sequentially:

Model A (Architect/Debugger): Qwen3.8 Flash MoE / Gyro-Coder TQ2\_0 (\~18.6 GB GGUF, 96k context).

Model B (Coder): Swift-ATX-27B (Qwen 27B fine-tune) in IQ4\_XS-M (14.3 GB) or Q5\_K\_M (18.7 GB) with MTP 5 and TurboQuant KV.

The plan:

Load Model A into VRAM -> Generate architecture plan.

Evict Model A to System RAM, copy Model B from System RAM into VRAM -> Generate code.

Evict Model B to System RAM, reload Model A from System RAM to VRAM -> Audit/debug.

The Physics vs. The Reality

Theoretical PCIe Transfer: At \~12 GB/s over PCIe 3.0 x16, transferring 14 to 18 GB from RAM to VRAM physically takes only 1.2 to 1.5 seconds.

Real-World Latency with llama.cpp / llama-server: 25 to 30.5 seconds per swap!

Why it takes \~29s instead of 1.5s

We profiled the execution and audited the llama-server router implementation (tools/server/server-models.cpp with --models-preset and --models-max 1):

Process Teardown: llama-server router mode is an HTTP process supervisor, not an in-process weight swapper. Switching models terminates the active child process and spawns a new executable instance.

Cold CUDA Context Overhead: Every new process incurs driver initialization (cuInit, primary context creation, stream pools, and CUBLAS handle creation) taking 3 to 5 seconds.

Windows Page Faults (mmap): Even though the GGUF pages already sit in Windows Standby RAM, the newly spawned process has empty Virtual Memory Page Tables (PTEs). Touching 15+ GB triggers millions of soft page faults.

VRAM Re-allocation: Allocating the 24 GB virtual pool and clearing unified KV cache buffers takes several seconds on startup.

What is Needed

To achieve the theoretical \~1.5s PCIe transfer, the runtime must:

Keep a single persistent process with a warm CUDA context that never dies.

Keep the idle model in page-locked/pinned host memory (cudaHostAlloc / cudaHostRegister / VirtualLock).

Swap weights via a direct, asynchronous DMA copy (cudaMemcpyAsync) straight into a pre-allocated GPU tensor buffer, then repoint the graph pointers.

The Question to the Community

Has anyone found or built an engine/wrapper that can hot-swap full model weights between Pinned RAM and VRAM inside a live CUDA context without restarting the server process?

Can engines like Strata (Niko1221/Strata), NInfer (Neroued/ninfer), or custom PyTorch/vLLM memory pool forks support swapping two completely different model architectures (e.g. Recurrent GDN MoE vs Dense Attention Transformer) in <2 seconds?

Is there any PR or trick in llama.cpp to retain VRAM allocation/context and just reload weights via memory-mapped DMA?

Any insights, forks, or technical solutions would be greatly appreciated!
```



document fetched 2026-11-01 00:38 · canonical https://reddit.com/r/LocalLLaMA/comments/1xk6qbw/fast_model_swapping_ram_vram_2s_on_a_single_24gb
