← sources

What happens when the data you process is much much larger than the available GPU Memory?

source: reddit (reddit:developersIndia)published 2026-08-31 17:00ingested 2026-09-01 02:36

retained text

This was the question I started my research journey \~4 years ago.

I was working on GPU algorithms for graphs with billion edges (500 600GBs) while having access to just 40GB GPU. The obvious answer was "use a larger GPU", except, ahem, govt institute, ahem budget. 🙂

That's where I started my exploration on how to utility the available resources better. What initially started as a GPU Programming / engineering problem quickly became an algorithm + architecture + memory + data-movement problem.

Looking down the lane, which felt like a niche problem half a decade ago is much more relevant today with the RAM shortage, memory prices growth of LLMs in general.

And somewhat unexpectedly, this simple question ended up becoming the foundation for 6 published papers.

Edit: As someone commented, the workaround / solution is much more relevant than the problem, so here is the approach that we followed.

The classic textbook solution would be:
- OOM and fatally stop or reap processes to save yourself  not relevant here). 

- swap/read from disk. 

- shrink your input size to fit in your buffer. 

Yeah, followed last two approaches, i.e. shrinking the working set to fit GPU memory then overlapping data movement with computation, and then merging the partial outputs. One other obvious solution (those who work with GPUs know) is Unified Memory, but in our case it didn't really work either: page faults, PCIe transfers, irregular graph access patterns, and limited GPU memory meant the performance could degrade badly.

Fitting the data was relatively straightforward; the real bottleneck was figuring out what to do with each batch without killing parallelism, and how to efficiently merge the results. That eventually became an algorithmic problem rather than just a memory-management problem, leading us to redesign the algorithm. 

document fetched 2026-09-01 00:24 · canonical https://reddit.com/r/developersIndia/comments/1w3jrf6/what_happens_when_the_data_you_process_is_much