loading
loading
Meta published a detailed look inside Prometheus today, the gigawatt-scale AI supercluster it is building in New Albany, Ohio. The first phase is complete and has trained the Muse family of foundation models, including the Muse Wonder models, and Meta says the site is being scaled toward roughly 1 million GPUs. Two years ago, the same campus was a conventional data center serving web traffic. The post, from Meta engineers Kevin Lee, Yashar Bayani, Ahmed Sharif and Yee Jiun Song, covers the hardware, the network, and the reliability work that keeps training jobs running. Meta has also disclosed some of the most interesting data around running large training jobs: the reliability figures. Meta reports more than 85 percent goodput on Muse Wonder pre-training runs and more than 90 percent on reinforcement learning runs, with a wall-clock mean time between interruptions of about 47,000 GPU hours during those runs. Firmware fixes rolled out since then have pushed MTBI past 54,000 GPU hours. Meta’s stated target is two interruptions per 8,000 GPUs per day, which works out to roughly 100,000 GPU hours between interruptions. Two NVL72 Generations in Production, Vera Rubin Next Prometheus runs two NVIDIA GPU platforms today, both built on the NVL72 rack architecture where 72 GPUs share one NVLink domain, with a third based on Vera Rubin NVL72 arriving soon. The first, which Meta calls Catalina, uses the GB200 NVL72 in the Ariel variant: a two-rack configuration with 36 GPUs per rack, built on Meta’s Open Rack V3 high-power rack design contributed to the Open Compute Project. Each Blackwell GPU pairs with a Grace CPU and two ConnectX-7 400G NICs, a 1:1 CPU-to-GPU ratio that gives every GPU 800G of scale-out bandwidth. Meta says it picked Ariel because the GB200 NVL72 scaled FLOPS, HBM bandwidth, scale-up bandwidth, and domain size well past the H100, while standard scale-out would have stayed at 400G per GPU, and it expected that imbalance to cost performance. Per Meta, a Catalina pair carries 72 Blackwell GPUs with 13.4TB of HBM3e (192GB per GPU), 720 PFLOPS of peak FP8 training compute (1,440 PFLOPS at FP4), 130TB/s of NVLink bandwidth inside the domain, 576TB/s of aggregate memory bandwidth, and draws about 120kW across the two racks. The second platform, Clemente, moves to the GB300 NVL72. HBM per GPU rises 1.5x to 288GB, dense FP4 throughput rises by a similar factor, and attention acceleration roughly doubles. ConnectX-8 SuperNICs deliver 800G per GPU natively, so the scale-out headroom Meta engineered into Ariel becomes the baseline, and a 2:1 GPU-to-CPU ratio lets the full 72-GPU domain fit in a single rack. Meta lists 20.7TB of HBM3e, more than an exaFLOP of peak FP4 compute, 130TB/s of NVLink bandwidth (1.8TB/s bidirectional per GPU), about 135kW per rack, and a 5 to 10 percent performance gain over the GB200 NVL72. Clemente deploys with either Meta’s modular air-assisted liquid cooling or facility liquid cooling in its newer AI-optimized buildings, and Meta says serviceability improves with less cabling. Meta’s next platform, San Miguel, will bring Vera Rubin NVL72 into the same network fabric. Meta’s figures for that rack are 288GB of HBM4 per GPU at 19.2TB/s, 2.52 exaFLOPS of peak FP4 training compute and 1.26 exaFLOPS at FP8, 216TB/s of NVLink bandwidth (3TB/s per GPU), about 1,500TB/s of aggregate memory bandwidth, fully liquid-cooled compute trays at more than 170kW per rack, and a 10 to 20 percent gain in performance per watt over the GB300 NVL72. CoreWeave put Vera Rubin NVL72 into production earlier this month. A Dedicated Ethernet Back End Spanning Buildings Training traffic runs on a second data center network over Ethernet, separate from the front-end network that carries ordinary service traffic. The design follows the shape of a training job: data-parallel replicas keep their latency-sensitive, bandwidth-heavy collectives inside a building fabric, while the more tolerant traffic between replicas crosses an aggregation layer that links buildings. Meta built the same hierarchy into its earlier 24,000-GPU building-scale clusters and its 100,000-plus GPU regional cluster before stretching it to Prometheus. Each building fabric connects up to 20,736 GPUs. Meta’s first design was a Disaggregated Scheduled Fabric that used sub-packet multipath routing, in-fabric reordering, virtual output queues, and credit-based flow control to keep the few large, long-lived flows of a training collective from colliding. As NICs gained the ability to reorder packets at the endpoint, Meta moved to a Non-Scheduled Fabric built on shallow-buffer 51.2T switches: Minipack3 with dynamic load balancing, and Minipack3N, which uses the NVIDIA Spectrum-4 ASIC with Adaptive Routing on ConnectX NICs, with both running Meta’s FBOSS network stack. Above the buildings sits the Backend Aggregation network, a multi-layer hub-and-spoke design built on Arista 7800-series chassis with deep-buffered Broadcom Jericho3 silicon. It spans 2km to more than 100km and delivers multiple petabits per second between region pairs. Meta says running pre-training across those distances took a dedicated optimization effort with Meta Superintelligence Labs because of speed-of-light latency. The plant behind it includes millions of miles of fiber strand, tens of thousands of switches, and millions of optics, from 2x400G FR4 LITE pluggables inside buildings to 800G-ZR coherent optics for links beyond about 10km. 64 of Every 72 GPUs Do the Work Moving from the H100 generation’s eight-GPU NVLink domain to NVL72 multiplied the failure blast radius by nine, and Meta now plans capacity around the 72-GPU rack. Because training parallelism maps best onto power-of-two group sizes, each NVL72 domain runs a 64-GPU slice and holds the other eight GPUs as in-rack spares. If a GPU fails, a spare takes over, and the job restarts from its last checkpoint without waiting on a repair. For whole-rack faults, Meta keeps about 2 percent additional racks in reserve. For a 32,768-GPU training run: 523 NVL72 racks, or 37,656 GPUs, about 13 percent above the active footprint, with 11 points of that coming from the 64-of-72 slice geometry. The signal integrity burden behind that reliability is considerable; a single NVL72 domain carries 5,184 tuned differential-pair links at 200Gbps, each crossing GPU SerDes, compute tray PCBs, cable cartridges, and switch tray interfaces. Two recovery paths handle the faults that do occur. When a failure can be isolated to one data-parallel replica, Meta’s parallelism-aware fault tolerance takes that replica offline while the rest of the cluster repeats the step and keeps training, and the slice restarts on spare hosts and resynchronizes. Larger faults send the whole job back to its last checkpoint, so Meta checkpoints often, saving model state including gradients and optimizer data asynchronously to multi-tier storage with delta checkpointing and in-memory staging to limit the hit on throughput. A 99 Percent Drop in Interruptions For all the progress those numbers show, Prometheus did not start out stable. In mid-2025, during the initial phase, interruption rates peaked at nearly 1,000 events per 8,000 GPUs per day. A year of cross-functional work has cut that by 99 percent, and the cluster now sustains multiple training jobs of tens of thousands of GPUs each in parallel. Meta credits a co-engineering model with NVIDIA in which NVIDIA teams sit at the New Albany site, share telemetry, and work joint triage sessions that decompose every halt by root cause, with Meta owning scheduling and infrastructure fixes and NVIDIA covering firmware, signal diagnostics, and memory handling. NVLink signal integrity was the first bottleneck on GB200. With more than 600,000 NVLink connections in the cluster, marginal links triggered frequent recovery cycles, so Meta added continuous telemetry, automated flagging, and L11 diagnostic burns to find weak links before production. Memory faults in HBM and LPDDR took over as the leading cause as GB300 arrived with larger memory, and the response there is refined HBM firmware settings, memory scrubbing, an improved LPDDR error correction algorithm, and predictive retirement of degrading devices. Switch health is monitored at millisecond resolution, and link events in the aggregation layer are rerouted in hardware without interrupting GPU traffic. Operating hundreds of thousands of first-generation accelerators turned hardware maturation into its own discipline. Meta’s firmware automation can push driver and firmware fixes to more than 100,000 GPUs in a few hours, and the P90 time to upgrade a single pod has dropped from more than seven hours early in the GB200 introduction to two hours. Power delivery stress testing looks at how PSU modules and voltage regulators behave under the 1,400W load of a single GPU, and every repaired system sits in quarantine until it passes validation. Meta’s stability sprint in March synchronized fixes across the stack. Power Transients and a Mixed Cooling Fleet Synchronous training moves power in lockstep, and tens of thousands of GPUs entering a compute phase or a communication phase together swing cluster load by 30 to 40 percent, and starting or pausing a large job can shift tens of megawatts in seconds. Dynamo Dimmer, Meta’s cluster-wide power management framework, caps GPU power gradually as the cluster approaches its electrical ceiling so jobs keep running. Rack-level battery backup modules absorb spikes, capacitor trays smooth peaks, facility-scale battery energy storage sits behind them, and Meta coordinates with utilities on the load profile. Cooling varies by building because Prometheus was assembled quickly across free-air data centers, rapid-deployment structures, leased capacity, and a purpose-built AI building. The older spaces use modular air-assisted liquid cooling; the AI-optimized building uses facility liquid cooling in a closed loop, which Meta describes as the most efficient way to cool GPU servers and the most water-efficient. Meta has shared designs such as IcePack, its liquid-cooled network rack, through OCP. For a look at what a far smaller GB300 NVL72 liquid-cooled deployment involves, see Giga Computing’s GAIFA test bed. Hyperion Comes Next Meta will keep expanding Prometheus with a mix of GB300 NVL72 and Vera Rubin NVL72 racks, with Vera Rubin arriving through the San Miguel platform on the existing network fabric. The lessons from New Albany are feeding Hyperion, the supercluster Meta is designing from scratch in Richland Parish, Louisiana, with closed-loop liquid cooling, full grid connectivity through Entergy, and next-generation network designs. Meta says it pays the full cost of energy at both sites and matches its use with new clean and renewable generation. Most of what Meta published is operational detail that hyperscalers usually keep to themselves: the spare-GPU configuration, the interruption rates, the firmware rollout times, and the reasons behind the Ariel configuration choice. The through-line is that at this scale the hardware is only the starting point, and the work that turns hundreds of thousands of GPUs into usable training capacity is in the network, the fault tolerance, and the telemetry. For enterprises looking to deploy AI at scale, this Meta deep dive is a good starting point to at least understand where issues may arise, so organizations can plan ahead across their entire data center estate.
document fetched 2026-10-24 11:22 · canonical https://storagereview.com/news/meta-prometheus-gigawatt-supercluster-1-million-gpus-gb200-gb300-nvl72