Microsoft’s “Clea” Maia 300 Inference XPU Is A Bandwidth Beast

extract

Microsoft’s “Clea” Maia 300 Inference XPU Is A Bandwidth Beast If everyone has an AI inference problem, with tokens still being too expensive to generate even with state of the art GPU and XPU accelerators, Microsoft probably has the biggest inference problem in the world. For one thing, running inference for OpenAI ChatGPT models as well as helping the AI model builder train its models is a big job. And then all of Microsoft’s worker productivity and back office applications are being infused with AI routines, and this is driving usage up the hockey stick handle at a rapid pace. That is why Microsoft is putting the pedal to the metal with its Maia family of custom, homegrown XPU accelerators, and letting customers know Microsoft is working on the problem is why the company has rolled out the feeds and speeds on its future “Clea” Maia 300 accelerator as the Open Compute Project Summit 2026 kicks off this week. The Maia 300 chip is a much-improved device, and it has a number of architectural changes that are designed to allow GenAI inference to run not only faster, but to scale up further so that larger agentic workloads, that consist of dozens to hundreds of models – implicitly through a mixture of experts foundation model, actually physically distinct independent models, or a mix of the two – working in concert and needing to share the same network and memory space to work efficiently and with low latency. Microsoft is no slouch when it comes to designing chips, and it often gets things right on the second or third try. The software and cloud giant dipped its toe in the water with the “Athena” Maia 100 XPU in November 2023. At the time, Microsoft was also unveiling its Cobalt 100 custom Arm server CPU, and it said very little about the architecture of either custom chip. The second generation “Braga” Maia 200, which was only unveiled in January of this year, is a much more serious XPU and is the one that has been ramping into production on the Azure cloud this year. In a separate announcement by Scott Guthrie, executive vice president in charge of Cloud and AI at Microsoft, he said that the “multigenerational Azure Maia program has become a foundational element of Microsoft’s AI infrastructure strategy,” and said that Maia 200 chips are doing inference for Copilot, helping the Microsoft AI team to develop new models, with GitHub Copilot and Microsoft Foundry shifting to Maia 200s by the end of the year. (GitHub Copilot was supported by Maia 100s previously.) The Maia 200s will run OpenAI’s various models, Microsoft’s own models, and various open source models through Microsoft Foundry. It is not clear how much of OpenAI’s GPT API service is running on Maia 100 or Maia 200 iron. The rumor mill is that Microsoft wants to get at least 1 million Maia sockets into the field, and Guthrie said in his statement that Maia would have “multi-gigawatt deployments” on the Azure cloud. A gigawatt of Maia 300s is around 575,000 devices, all in at 1.74 kilowatts for each accelerator and its portion of the system host, storage, and scale up networking. (Microsoft said a rack of Maia 300s burned about 167 kilowatts and has 96 XPUs interlinked.) This Braga chip had much more HBM memory capacity and bandwidth even as it trimmed back on the on-chip SRAM cache. Moreover, it sported a much improved AI Transport Layer (ATL) custom RoCE Ethernet implementation that sported 2.3X more bandwidth while scaling up the cluster coherence domain of Maia cluster by 2.7X to 6,144 devices while only increasing power consumption by 1.5X. By shifting from 6-bit MX6 data formats to 4-bit MXFP4 and boosting the number of compute cores by 1.5X while at the same time boosting the clock speed, the Braga chip had 3.4X more floating point oomph than the Athena chip it replaced. The Clea Maia 300 chip – really a chip complex in a socket, perhaps a system in a socket and not, strictly speaking, a system on a chip as Microsoft refers to it in its technical deep dive since it is actually a collection of seven chips and eight HBM stacks on an interposer – is pushing the envelope on all spec fronts, but is clearly focused on effective performance on real-world GenAI inference workloads and not on just cramming flops in a socket for speeds and feeds spec sheet superiority. Moreover, Microsoft has changed its ATL Ethernet network so it can cram more XPUs into a rack and more racks into a coherent domain to take on bigger agentic workflows. The Maia 300 “system in socket” and the racks of machines to make up pod and superpod clusters have a hierarchy of compute, memory, and networking to glue together aggregations of these capacities to bring them to bear on GenAI work. The Clea compute tiles are a little different from the ones comprising its Athena and Braga predecessors. All three XPUs have cores with a tile tensor unit (TTU) and a tile vector processor (TVP) as week as a tile data movement engine (TDMA) and tile control processor (TCP), plus L1 SRAM cache memory for these units to share. (How much was not divulged.) The Athena XPU organized four of these cores on a tile and sixteen tiles into a cluster (an in-socket cluster, that is) to get 64 cores in total. The organization of the Braga tiles and tile clusters was not divulged last January, but a paper was published on arXiv back in August that did and that has allowed us to update our salient characteristics monster table for the Maia lineup. The Braga XPU, as it turns out, has tensor units with matrices that are twice as large as the Athena unit, so it took a little less than half as many cores to get the throughput Microsoft pegged for the device than we thought back in January’s story. The Braga XPU has ten tiles per on-chip cluster and four clusters for a total of 40 cores. They run at 2 GHz, and only nine cores are active per cluster to produce yield on the 3 nanometer N3P process used to etch the chip by Taiwan Semiconductor Manufacturing Co. The Athena and Braga chips were monolithic designs, and close to the reticle limit of lithography machines. With the Clea XPU, Microsoft is moving to the 2 nanometer N2P process from TSMC and is also moving to a chiplet architecture that breaks out I/O and HBM controllers separate from the compute and SRAM cache elements. With Maia 300, there are four tile clusters, just like with the Maia 200, but each on-chip cluster has twelve tiles. There are also two elements, a reshaper and a tile synchronizer, that were added to the architecture with the Maia 200, but which we did not know about earlier this year, that help with the dataflow that spans the entire system, from the network down to the server node and all the way down to the tile. The SRAM capacities per tile (L1) and per cluster (L2) are not revealed. If I had to guess – and I do – there is around 400 MB of SRAM cache spread across the compute tile at the center of the Clea chip package and another 200 MB embedded in the I/O chiplets to help cache network data as it enters and compute data as it exits the package. There could be logic as well as SRAM embedded in the HBM controller dies, which are relatively large, as you can see. I am guessing that there is 200 MB of SRAM in the two I/O dies on the Clea package. Across the Clea complex, I reckon that there is about 166.2 TB/sec of aggregate SRAM bandwidth, which counterbalances the 18 TB/sec of HBM memory bandwidth for the eight HBM4 stacks on the device. The Clea XPU has 288 GB of HBM 4, which is presumably from SK Hynix as was the case with the Braga XPU. As for area and transistor count for the Clea package, you can make a rough estimate based on the fact that HBM memory sizes are known and we have images of the chip packages for all three Maia chips as well as their process nodes. I assume, by the way, that the HBM memory controller and I/O dies used in Clea are using 5 nanometer, not 2 nanometer, processes, which would make them perform better and also be more cheaply made. (Shrinking I/O circuits does not necessarily improve their performance, and it most definitely raises their costs and shrinks their yield.) It looks like the Clea package is about 1,980 mm2 in area, which is 2.4X times as large as I think the Braga package was, at 810 mm2 in area. Playing around with some numbers, I peg the transistor count at around 280 billion, give or take maybe 10 percent, not including the HBM4 memory. Take a look for yourself and make your own guesses: Here are the feeds and speeds for all three Maia XPUs, with estimates by me made in bright red bold italics as usual: Microsoft has not said who its chip shepherd is for the Maia 300, but Global UniChip (GUC) of Taiwan, a close packaging partner of TSMC (which owns 35 percent of the company), was said to be the shepherd for Maia 100 a few years back. Everyone presumes that GUC is shepherding Maia 200 and Maia 300, although there were rumors two years ago that Marvell was trying to muscle in on the Maia line and, soon thereafter, there were rumors that Broadcom was trying to geta piece of the Microsoft custom silicon biz. As was the case with the Braga XPU, the Clea XPU interlinks four accelerator motherboards together and puts them into a server form factor. With Braga, the CPU hosts were outside of XPU racks. But with the Clea XPUs, the host server – which we presume in both cases is a 132-core Cobalt 200 series Arm server CPU designed by Microsoft and shepherded in some fashion by Arm itself – is pulled inside of the CPU node. In either case, we assume the CPUs and XPUs are linked with PCI-Express switches, not Ethernet ports. But, maybe not. Microsoft may pick another CPU chip shepherd now that Arm is selling its own AGI processor. Hence the rumors about Broadcom last year. Here is what the Maia 300 server looks like: If you squint, it looks very similar to the Maia 100 and Maia 200 servers, with the CPU pulled inside the chassis. But there are more differences than meet the eye. All three Maia generations are wired together differently. Here is how the Maia 100 RoCE Ethernet wired accelerators to each other within a node and then out to a two-level network to each other across a rack: Nine of the twelve 400 Gb/sec links were used to link a quad of Maia 100s very tightly to each other, and three of the remaining links were used to hook into a three-plane Ethernet fabric that was the first tier of the network (which is T0 of course). This first tier could interlink 32 accelerators to each other, and a second tier of Ethernet switches were used to scale the system out to 576 nodes and 2,304 Athena XPUs in a single coherency domain. With the Maia 200, the I/O portion of the chip implemented 28 Ethernet ports running at 400 Gb/sec for a total of 2.8 TB/sec of aggregate, bi-directional bandwidth. Here is how most of those ports were used: There are four switch planes in this ATL 1.0 network, so that burns four ports per accelerator. Interestingly, Microsoft once again tightly coupled a quad of Maia 200s into a shared memory system with lots of bandwidth between the devices, with each Maia 200 connected to others with seven ports excepting one per chip that was used to link into the first plane (Switch Plane 0) in the diagram. That’s a total of 20 ports for linking the XPUs to each other and another four to link each XPU into each of the four planes for a total of 24 out of the 28 total ports. I have no idea where the other four ports are used. Perhaps they link the memory of the CPU host coherently with the accelerators using the ESUN or UALoE protocols? What I can see is that a dozen trays of four Braga XPUs comprises a rack with 48 devices, and up to 128 racks can be interlinked into the coherency domain with 128 first tier (Tier 0) switches and 32 second tier (Tier 1) switches to span 6,144 XPUs in a single currency domain. These Ethernet switches have 51.2 Tb/sec ASICs and have 128 ports in total running at 400 Gb/sec. Here is the important thing: the Braga device was set up to share all of its HBM memory across all of its compute tiles and the Braga system board tightly coupled four of these XPUs to also share their memory in a NUMA fashion. With the Maia 300, a pair of compute tiles is tied to a pair of HBM4 memory banks in a NUMA-ish way. Meaning, memory is not shared across all tiles and all HBM4 stacks on the socket explicitly. This will make routines that fit into that memory bank run faster, which is important for GenAI inference. As you will see soon in my analysis of the OpenAI “Jalapeno” AI accelerator, which I am finishing up now, this is a tack that OpenAI and Broadcom did in the Jalapeno design, too. As I have been warned by some techies, I over-value coherence. Apparently Microsoft did, too. So, with the Maia 300, only two accelerators are linked to each other, instead of four, as well as there being four distinct memory HBM4 regions within the Clea package. Here is what it all looks like, with the ATL 2.0 network lashing it all together: Microsoft has a next generation of liquid cooling for the Maia 300 racks, and is moving from a 2U form factor to a 1U form factor, which allows it to double up the number of XPUs per rack to 96 by having two dozen servers instead of a dozen. Across 128 racks, that gives you 12,288 accelerators in a single system image. Like this: That is the biggest coherency domain we have heard of yet across the big cloud builders, and a heck of a lot larger than the 72 or 144 GPU coherency domains that Nvidia and AMD are doing in a coherent fashion across their rackscale systems, or even the 576 or 1,152 multi-tiered NVSwitch coherent fabrics that Nvidia is promising for production environments in the future. Google’s “Sunfish” TPU 8t training chip (which is not yet shipping) tops out at 9,600 TPUs in a coherency domain, while the “Zebrafish” TPU 8i inference chip has a “boardfly” topology that peaks at 32 in a boardfly (a dragonfly topology of eight dragonfly nodes with four TPUs per node) and 1,152 in a group of 36 boardflies linked to each other over the “Apollo” optical circuit switch that is the backend for the Google network. To save you from having to do the rack math, Microsoft did it for you here: These are raw feeds and speeds, of course, and are not necessarily what happens when real applications run. What Microsoft’s top techie brass – Rani Borkar, president of Azure Hardware Systems and Infrastructure, and Saurabh Dighe, corporate vice president of Systems Planning and Architecture – said in their deep dive is that Maia 300 with 96 XPUs in a rack delivers up to 7X the “token throughput” performance of a rack of Maia 200s with 48 XPUs in a rack. You get 36 percent more flops moving from Maia 200 to Maia 300, and then another 100 percent from the XPU count doubling in the rack. The real difference here has to be architecture, by coupling the XPUs and nodes in the system differently, having more HBM and SRAM memory and a lot more bandwidth at all stages of the calculations. And the dataflow that moves information around the system so it can be processed and created is no doubt a big, big part of the improvement. We look forward to the paper that will come out a year from now telling us how Microsoft did it. Here is the metric that Microsoft is likely most proud of: You get that 7X performance improvement with a 2.5X improvement in performance per watt. It is easy – well, relatively easy – to get more performance by throwing more power at things. It is another thing to do it while increasing the power efficiency.