loading
loading
ARCHITECTURE Breaking the KV cache memory wall using Dynamo and Liqid CXL shared memory Micron has found that KV cache memory wall limitations on a GPU server can be substantially reduced by using a Liqid CXL-accessed memory pool to enlarge the DRAM cache. A Micron team is building a GPU cluster for the US Department of Energy’s Pacific Northwest National Laboratory (PNNL) to run next-generation, large-scale scientific, AI, HPC and data-intensive workloads. The cluster is called Great Abaco, and Dr. Skyler Windh and his team at Micron developed a high-capacity shared memory system architecture for AI and HPC workloads. It uses CXL 2.0. The system is described in a 60-minute webinar and the background is this: as Large Language Model (LLM) inference scales across multi-GPU deployments, the Key-Value (KV) Cache becomes a major performance bottleneck. Each GPU typically keeps its own isolated cache, which can spill to host DRAM. When different GPUs process the same prompt prefix, each one repeats the same prefill computation. CXL fabric-attached memory (FAM) addresses this by providing multiple GPUs with a shared, byte-addressable memory pool, allowing other GPUs to reuse KV Cache entries computed by one GPU. Windh and his team built cross-host KV sharing into Nvida Dynamo and ran it on Micron’s Great Abaco cluster. Abaco is a shared-memory system architecture for AI and HPC, designed by Micron with Liqid as the memory fabric and orchestration partner. A single Abaco rack presents up to 160TB of DDR5 as an addressable pool that one or multiple hosts in the rack can claim, share, or release without a reboot and without re-cabling. Applications reach the pool through a standard file system interface, so most code runs unmodified. The cluster uses H200 NVL GPUs in a Liqid PCIe Gen5 chassis, with four Liqid CXL memory chassis forming one shared pool. On Llama3-70B with 90 percent shared prefixes, a 32 GB cache per server delivered 2.8× the throughput and 2.9× faster time to first token than DRAM of the same size. Hosts reusing each other’s KV blocks got 3× the throughput with half the memory, and CXL memory ran within 99 percent of host DRAM. Ie. CXL FAM performs on par with a host DRAM baseline. The Micron team integrated CXL FAM-based KV Cache sharing into an open-source multi-GPU inference framework, with: - A block-level cache manager that allocates in CXL memory - Sequence-hash-based cache discovery across GPUs - An asynchronous transfer engine that moves blocks between CXL and GPU memory without stalling the inference scheduler - A CUDA host-registration workflow that enabled direct memory access (DMA) between GPUs and CXL memory Liqid SVP for Solution Architecture Martin Cooper says this Micron work shows that shared CXL memory can lift GPU inference throughput 2.8× over DRAM. Liqid puts GPUs and CXL memory on one PCIe fabric, so memory scales separately from compute. The serving stack stays as it is: no vLLM patches and no changes to the GPU servers. A shared pool lets you right-size each server and buy the memory once. You can access the webinar 's 30-slide presentation deck here, and watch the whole webinar here - it's has many YouTube adverts though. You can download a transcript we created here if you wish, and read it along with the presentation deck. Download an Abaco technical brief here. Bootnote GFAM, referred to in the slides = Global Fabric Memory
document fetched 2026-10-27 02:55 · canonical https://blocksandfiles.com/architecture/2026/10/26/breaking-the-kv-cache-memory-wall-using-dynamo-and-liqid-cxl-shared-memory/5305130