loading
loading
Microsoft unveiled Azure Maia 300 today at the OCP Global Summit, the third generation of its in-house inference accelerator and the first built as a multi-chiplet part. Each accelerator pairs a TSMC N2P compute die with eight HBM4 stacks for 288GB of memory at 18TB/s inside a 950W TDP, and a Maia 300 rack holds 96 of them in a 167kW envelope. Microsoft claims 40% more tokens per megawatt than leading GPU-based systems and up to 7x the rack-scale token throughput of Maia 200. Production deployments begin in 2027 as part of what Microsoft describes as multi-gigawatt Maia deployments. Maia 300 Silicon: N2P Compute Die, Eight HBM4 Stacks, 32 Ports of 400G Maia 200 was a monolithic N3P die with six HBM3E stacks. Maia 300 splits the design into a main compute die on N2P, two I/O dies, and four memory dies, with the eight HBM4 stacks taking capacity from 216GB to 288GB and bandwidth from 7.1TB/s to 18TB/s. Peak compute moves less: 13.8 POPS at FP4 and 6.9 POPS at FP8, up from 10.1 and 5.1, so the generation is a 37% gain in FP4 compute against a 2.5x gain in memory bandwidth. Microsoft’s summary of the chip is 2.5x the bandwidth, 33% more capacity, and about 2x the HBM bandwidth per watt of Maia 200. The compute side is organized as four clusters of 12 tiles, each tile carrying a tensor unit, a vector processor, a reshaper, a DMA engine, and local SRAM, with a shared cluster SRAM and a network-on-chip above that. Microsoft calls the approach a software-defined dataflow architecture: software places data across those local memories, DMA engines, and the network. The SoC exposes PCIe Gen6 x16 to the host and 32 ports of 400Gb/s Ethernet through an on-die AI network controller, which is the scale-up fabric. On Maia 300, the memory organization moves from uniform access to a NUMA layout across the chiplets so data can be placed near the compute that uses it, and precision drops further with enhanced MXFP4 and hardware data type conversion, aimed at cutting the bytes moved per token for both compute and KV cache storage. Kernels can also use on-die SRAM as a software-managed scratchpad to keep traffic off HBM. Microsoft attributes the per-watt gain to those changes, since the TDP rises from 750W to 950W while the company claims more than 2.5x the performance per watt. 96 Accelerators per Rack, 12,288 in a Superpod A Maia 300 rack doubles the accelerator count to 96 and raises rack power from 64kW to 167kW, which works out to about 1.74kW per accelerator slot with networking, cooling, and power conversion included. Across the rack, that is 27.6TB of HBM, 1,728TB/s of aggregate HBM bandwidth, and 1,324 POPS of FP4 by Microsoft’s figures, against 10.4TB, 341TB/s, and 485 POPS for a Maia 200 rack. Bidirectional scale-up bandwidth per accelerator rises from 2.8TB/s to 3.2TB/s, and the two-tier Ethernet scale-up network with Microsoft’s custom AI transport now reaches a superpod of 12,288 accelerators, double the roughly 6,000 of the prior generation. Microsoft’s rack diagram shows jobs sharded across racks over the scale-up fabric with expert-parallel traffic between shards, which is the traffic pattern that large mixture-of-experts models generate at serving time. The company says the same communication model covers any accelerator in the superpod, so a 16-accelerator job and a job spanning four racks use the same semantics. Maia 300 Specifications Where Maia 300 Fits and What the Claims Rest On Maia 200 is now in production behind Microsoft Copilot and the MAI model team, with GitHub Copilot and Microsoft Foundry due to move onto it by the end of the year, and Microsoft says it is deploying across the United States, Europe, and Australia with up to 1.7x more tokens per watt than its GPU infrastructure for the built-in AI features in Teams, Copilot, and Cowork. “We will scale our Azure Maia family to multi-gigawatt deployments starting in 2027, making Maia the primary inference system for Microsoft products and services,” wrote Scott Guthrie, executive vice president of Cloud + AI, in announcing Maia 300. NVIDIA and AMD remain in Microsoft’s training fleet, and the company has not offered Maia as an instance type Azure customers can rent. The headline efficiency claim, 40% more tokens per megawatt than leading GPU-based systems, does not name the GPU, the model, or the serving configuration, and because Maia runs only inside Microsoft, nobody outside the company can test it. The published specifications do support the design: per-accelerator memory bandwidth rose 2.5x while FP4 compute rose 37%, and the rack delivers 5.1x the HBM bandwidth of a Maia 200 rack on 2.6x the power. That is a chip built for the decode phase of long-context and agentic inference, where the bottleneck is feeding the arithmetic units, and it indicates how a hyperscaler sizes an inference part when it owns the workload end to end. We covered the Maia 200 launch in January, and Microsoft says more detail is coming at Ignite in November.
document fetched 2026-10-13 02:56 · canonical https://storagereview.com/news/microsoft-maia-300-hbm4-n2p-chiplet-96-accelerator-rack