Memory Is The Lynchpin Of The IT Industry, And Doubly So For AI

retained text

STORE
Memory Is The Lynchpin Of The IT Industry, And Doubly So For AI
For as long as most of us have been watching high-end commercial and technical computing, the compute part of the system – the CPU for five decades then also GPUs for the past decade and a half – has been, without a doubt, the focal point of the system. Memory, storage, and I/O were always important, but they were secondary.
Budgets for systems and clusters were focused on maximizing compute, and these other factors were often secondary. And to be honest, organizations sometimes skimped on memory and I/O to the detriment of driving CPU or GPU utilization.
With the advent of machine learning and its evolution into generative AI and then agentic AI, memory has come front and center and has eclipsed compute as the control point in most of the IT sector. This is a bit disorienting for many of us, not so much in the concept of the change but in the rapidity of the magnitude of the change.
Here is the latest chip revenue breakdown from Gartner that shows memory versus non-memory sales, and flash is considered memory as far as the market researcher is concerned:
As you can see, the semiconductor market overall is on a Moore’s Law curve of sorts, doubling revenues every year. Call it More’s Law, if you will. The nearly doubling happened between 2025 and 2026, but as you can see, the forecasts show the growth rate decelerating between 2026 and 2027 – something we have been saying was inevitable since the GenAI boom. At some point, competition kicks in, and at another point, people have enough capacity to do what they need to do.
I estimated the breakdown between DRAM and NAND flash, which you can solve given the growth rates that Gartner gave in its statement for 2026 for these two categories. I assume the other types of memory are worth about $4 billion or so a year and are growing at 2X global gross domestic product, which legacy components and platforms tend to do.
I also added in Micron’s latest total addressable market forecast for HBM memory so we can back out other kinds of DRAM – DDR4 and DDR5, LPDDR4 and LPDDR5, and so on – from the mix. As you can see, the implied other DRAM revenues will dwarf the sales of HBM, probably for the foreseeable future.
The demand for DRAM main memory, HBM stacked memory, and flash storage – which commercial companies have adopted instead of disk drives as they try to “scale in” the performance of their workloads – is off the charts, and not just because GenAI is a memory and flash hog. We are in a massive upgrade cycle for systems, which need to be upgraded because machines that are five, six, or seven years old sitting out there in the server fleets of the world need to be upgraded. And to save space and power for AI systems, companies are going for near-maximum density of cores, and that is in turn driving demand for high capacity DRAM and flash. The traditional IT sector – transaction processing, data warehousing, data analytics, and web infrastructure – finds itself competing against the tech titans and their humungous GenAI aspirations for flash and DRAM memory. And DRAM supplies are tightening as memory makers shift their DRAM chip capacity to making HBM stacked memory for AI accelerators to chase higher revenues (but not necessarily higher profits).
This has made DRAM prices go bonkers. Back in November 2022, when OpenAI’s ChatGPT started the GenAI boom, DDR4 memory cost around $3 per GB for servers and DDR5 memory cost around $4 per GB. (These prices are averages across midrange capacities. Denser memory costs a hell of a lot more per GB, and skinnier memory costs considerably less.) A 30 TB TLC enterprise SSD cost somewhere around 10 cents to 13 cents per GB. Four years later, the street price of DDR4 server memory is from 2X to 3X more expensive at the street price, DDR5 server memory is anywhere from 9X to 13X higher, and that 30 TB TLC SSD will run you 6X to 7X more. Some of that change, by the way, was due to a memory and flash price crash that hit in late 2021, due to the oversupply issues that have plagued the memory and flash businesses for many decades.
And by the way, disk drive prices have skyrocketed, too. A 30 TB nearline disk drive costs about 2.5X over the same span, and if you think you can buy cheap disk, forget about it because the hyperscalers and cloud builders have contracted the vast majority of disk drive supplies from Seagate, Western Digital, and Toshiba, the three remaining disk drive makers.
HBM is a little different. First of all, there are three makers of HBM stacked DRAM memory – Samsung, SK Hynix, and Micron Technology – and there are no spot prices because the market is not open but dedicated to two GPU makers (Nvidia and AMD) and XPU makers and/or their chip shepherds. So pricing information is a little sketchy.
Back when HBM2 debuted in 2017, the rumor was that it cost around $20 per GB, but this settled down to around $8 per GB two years later. With HBM2E, which provided a bandwidth bump, it was around $10 per GB, rising to around $12 per GB for HBM3, which had more capacity per stack and more bandwidth, and the HBM3E with a bandwidth boost cost around $13.50 per GB. The HBM4 that will start shipping in the next generation of GPUs and XPUs, it is rumored to cost around $16 per GB – the rumor is actually that Nvidia’s contract price is $560 to $600 per 36 GB stack. For some reason, some people are expecting for HBM4E, which has a substantial bandwidth bump, to cost twice as much per GB as HBM4. I am not sure I believe that, given the trend data above for HBM2 through HBM4 memory. I can see $20 per GB, but $36 per GB is tough to fathom, even with the benefits.
Here is the thing to remember: The cost per GB of finished HBM parts is only up 1.6X since the GenAI boom started, while the number of DRAM chips that get tossed into the garbage because of perfectly normal but below 100 percent yields happen in manufacturing is increasing with each passing generation. The idea out there is that HBM makers will raise the price on HBM4E in 2027 to get the profitability of HBM more in line with standard DRAM.
By the way, these HBM costs above are not what end users pay to get the HBM memory on their devices, unlike the DRAM and flash prices above. The cost of the HBM to users as they buy GPUs and XPUs is much higher than the price that the vendors pay to get, as I explained in He Who Can Pay Top Dollar For HBM Memory Controls AI Training back in February 2024. The HBM memory probably represents half the cost to under users of a GPU from Nvidia and AMD, and is probably closer to 65 percent of the cost of an XPU accelerator made by the hyperscalers, cloud builders, or AI model builders now that OpenAI has entered the fray with its “Jalapeno” AI accelerator.
Here’s the thing. I have heard all the rumors about how GPU and XPU makers (and designers) are trying to do more with the HBM they can get allocated. Some are sticking with more stacks of HBM3E instead of moving to HBM4, maintaining most of the memory bandwidth – important for the decode part of inference – even as they sacrifice some of the memory capacity.
Nvidia is rumored to be backing away from its initial plans for the “Rubin Ultra” GPU coming in 2027, which was supposed to have four reticle-limited GPU chiplets and up to 1 TB of HBM4E memory. AMD is holding at 488 GB for its “Altair” MI455X GPU accelerator, while Nvidia is said to be considering going as low as 192 GB on some future Rubin and Rubin Ultra designs.
What seems more likely is that Nvidia will move the HBM controller off the GPU chiplet and onto the customizable base die that is possible with HBM4 and HBM4E designs – what Nvidia calls NVHBM but which is in no way unique to Nvidia. Others will have custom base dies for their HBM stacks, and more than a few will embed certain mathematical functions in there like processor in memory or PIM has been doing for a decade (without any market traction) to make some of the stuff happen in memory instead of being dragged across the memory channel into the GPU or XPU.
Here is why these custom base dies in the HBM stack matter, as Nvidia explains: “By integrating the memory controller into the 3D HBM stack instead of the XPU, NVHBM delivers up to 30 percent greater memory bandwidth and 15 percent lower HBM power consumption, and frees up to 25 percent more area on XPU compute die compared with standard HBM4E.”
If I was a betting man, I would put my chips on the idea that Rubin Ultra will make use of that extra 25 percent of freed-up chip area per chiplet to add more performance across two chiplets and maybe deliver 512 GB of HBM4E memory, maybe 384 GB with the same bandwidth. The original Rubin Ultra had four stacks per GPU chiplet, which were sixteen chips high at 4 GB a pop to reach that 1,024 GB capacity and, given the specs on the Samsung stacks, a whopping 58 TB/sec of aggregate memory bandwidth. By cutting down the stack heights and slicing the chiplet count in two, you can maintain 28 TB/sec of bandwidth and have 256 GB (eight high), 384 GB (twelve high), or 512 GB (sixteen high) of capacity.
All GPU and XPU makers are looking at similar choices when what they really want is the maximum HBM capacity and bandwidth for their devices on the leading edge technology at a much lower price per GB.

document fetched 2026-10-09 02:51 · canonical https://nextplatform.com/store/2026/10/08/memory-is-the-lynchpin-of-the-it-industry-and-doubly-so-for-ai/5301841