loading
loading
- With the recent release of the Muse family of foundational models, we want to take you inside the infrastructure that has made it all possible. - The first phase of Prometheus, our gigawatt-scale supercluster, is already complete and we’re working toward scaling it to comprise ~1 million GPUs. - Prometheus also features technologies like closed-loop liquid cooling to help it operate efficiently. Right now, in New Albany, Ohio, Meta is undertaking its most ambitious AI infrastructure project to date. Prometheus, our gigawatt-scale supercluster, is an advancement in our infrastructure that is training the next generation of foundational models that will bring new AI experiences to billions of people. Two years ago, New Albany was a conventional data center site running web traffic. With advancements in hardware, networking and building design, we reworked this site to be home to training programs for frontier models. We’re building the supercluster in phases. The first phase has already been completed and was used to train our Muse foundational models, including the Muse Wonder models, our new frontier models that deliver state-of-the-art performance on personal and professional agentic tasks. Large training jobs require everything to work optimally to sustain their performance; an issue on a single host can slow down or stop the entire job, lowering its ‘goodput’. By focusing on improving our systems’ stability and reliability we were able to achieve more than 85% goodput for pre-training runs and more than 90% goodput for reinforcement learning (RL) runs of Muse Wonder. During these training runs, we were able to achieve an overall wall-clock mean time between interruptions (MTBI) of ~47k GPU hours. Since then, we have landed further changes across our fleet including firmware fixes to increase MTBI to more than 54K GPU hours, and we are constantly improving it. Now, we’re continuing to scale Prometheus with our next frontier models from Meta Superintelligence Labs in mind. The Prometheus supercluster is housed in Meta’s New Albany, Ohio, data center. The Supercluster: Unlocking New AI Capabilities Today, Prometheus comprises an array of hundreds of thousands of GPUs across multiple interconnected buildings, including traditional and AI-optimized data center buildings and rapid deployment structures, all tightly interconnected by miles of fiber and functioning as a single machine. This capacity allows us to pursue larger models and longer context windows than were previously practical. At full scale, Prometheus will comprise roughly 1 million distributed GPUs, all functioning in lockstep. Multiple Generations of AI Hardware Prometheus currently deploys two NVIDIA GPU platforms, with a third (based on NVIDIA’s Vera Rubin NVL72) arriving soon. Each represents a generational leap in performance and efficiency, leveraging our NVL72 rack architecture where 72 GPUs act as a single accelerator within the same network fabric. Catalina: NVIDIA GB200 Our first platform, Catalina, leverages the NVIDIA GB200 NVL72. We deploy the Ariel variant, a two-rack configuration in which 36 GPUs per rack are interconnected via NVLink to form a unified 72-GPU domain. Catalina provides improvements in raw compute and memory capacity compared to Grand Teton H100. It leverages our Open Rack V3 (ORv3) high-power rack (HPR) structure, an open hardware design contributed to the Open Compute Project (OCP). The Ariel variant of GB200 NVL72 pairs each NVIDIA Blackwell GPU with an NVIDIA Grace CPU and 2x NVIDIA ConnectX-7 400G NICs at a 1:1 CPU-to-GPU ratio, letting us deploy 800G of I/O per GPU earlier than the rest of the industry. We chose Ariel because the GB200 NVL72 scaled up nearly every other dimension relative to the NVIDIA H100: FLOPS, HBM bandwidth, scale-up bandwidth, and scale-up domain size, while scale-out bandwidth per GPU would have stayed at 400G. We expected that imbalance to cost performance, so we opted for a configuration that delivers 800G of scale-out per GPU. This also gave us more uniformity of spec as the region expanded over time and 800G scale-out became available. The Catalina GB200 NVL72 features: - 72 NVIDIA Blackwell GPUs with 13.4 TB of HBM3e (192 GB per GPU). - 720 peak PFLOPS of FP8 training compute (1,440 PFLOPS at FP4). - 130 TB/s peak NVLink bandwidth within the domain. - 576 TB/s peak aggregate memory bandwidth. - ~120 kW total system power across the two-rack pair. Clemente: NVIDIA GB300 NVL72 Our second platform, Clemente, moves to the NVIDIA GB300 NVL72 and improves on Catalina at both the GPU and the platform level. GB300 raises HBM capacity per GPU by 1.5x, from 192 GB to 288 GB, and dense FP4 throughput by a similar factor, while roughly doubling attention (softmax) acceleration. At the platform level, scale-out is no longer the constraint it was on the GB200 NVL72: NVIDIA ConnectX-8 SuperNICs deliver 800G per GPU natively, so the capability we deployed early on Ariel becomes the baseline. That, together with a 2:1 GPU-to-CPU ratio, allows Clemente to pack the full 72-GPU domain into a single rack, streamlining our physical deployment. Clemente can be deployed with our modular Air-Assisted Liquid Cooling (AALC) or use Facility Liquid Cooling (FLC) in our newly designed data centers. Serviceability also improves substantially, with reduced cabling. The Clemente GB300 NVL72 features: - 72 NVIDIA Blackwell Ultra GPUs with 20.7 TB of HBM3e (288 GB per GPU — 50% more than the GB200). - Over 1 exaFLOP of peak FP4 compute. - 130 TB/s peak NVLink bandwidth (1.8 TB/s bidirectional per GPU). - ~135 kW rack power. - 5-10% performance improvement over the GB200 NVL72. Connecting the Supercluster In traditional web services, the data center network transports requests and responses between services whose instances typically don’t require high bandwidth, high performance, and tight, lock-step coordination across many servers. Large-scale AI training requires all these, meaning the network that connects them is now an important part of AI workloads. To best address this, in addition to the standard data center network connecting servers, several years ago we added a second dedicated data network based on Ethernet, called the Back-End (BE) network, to enable direct GPU-to-GPU communication both within and across data center buildings. Large pre-training jobs partition the training into data-parallel model replicas, creating a hierarchy in network usage: Latency-sensitive and bandwidth-sensitive operations concentrate within a replica, while traffic that is more tolerant of higher latency and less bandwidth is scheduled between replicas. Our network is designed to match this hierarchy, with low-latency, high-bandwidth fabrics within buildings and an aggregation network connecting the building fabrics. We work closely with Meta Superintelligence Labs to ensure the parallelism design, traffic patterns, and communication algorithms and software are optimized to minimize exposure of any network communication. This keeps the GPUs as busy as possible and not waiting on the network. The Meta team has employed these networking concepts in earlier building-scale, 24K GPU clusters and our region-scale, 100K+ GPU cluster over the last several years, and we took them to a new level of scale and distance in Prometheus. Network Fabrics At the building level, each Ethernet-based fabric interconnects up to 20,736 GPUs through a high-bandwidth, low-latency multi-layer topology. Our initial design used a Disaggregated Scheduled Fabric (DSF) to mitigate the network congestion caused by training collectives that typically have a small number of large, long-lived network flows that collide under traditional load-balancing techniques; it does this by utilizing sub-packet-level multipath routing and packet re-ordering in the fabric along with virtual output queues and credit-based end-to-end flow control. As Ethernet NIC technology advanced, enabling packet re-ordering at the endpoint, we were able to update to a Non-Scheduled Fabric (NSF) utilizing shallow-buffer 51.2T Ethernet switches, such as Minipack3 with Dynamic Load Balancing and the NVIDIA Spectrum-4 switch ASIC-based Minipack3N with NVIDIA’s Adaptive Routing on ConnectX NICs. Both DSF and NSF are based on Meta’s FBOSS networking stack, giving us the control to rapidly iterate on and optimize the fabrics in Prometheus, and the visibility to operate these clusters at scale. Aggregation The Backend Aggregation (BAG) network is a multi-layer hub-and-spoke network that connects building-level fabrics across the Prometheus cluster, spanning 2 km to over 100 km. It is built with Arista 7800-series chassis running deep-buffered Broadcom Jericho3 silicon, delivering multiple Pbps of bandwidth per region pair. Running pretraining over this network required a dedicated optimization effort with Meta Superintelligence Labs to ensure training was not slowed down, especially given speed-of-light constraints at these distances. Pulling this all together, the complexity of the Prometheus network involves millions of miles of fiber strand, tens of thousands of switches, and millions of optics — from 2×400G FR4 LITE short-reach, low-cost pluggables inside the buildings to 800G-ZR coherent optics for links beyond ~10 km. Engineering Reliability at Every Level At Prometheus’ scale, every component needs to be monitored for reliability. A single fault anywhere can halt an entire training job and waste valuable compute resources. Maintaining reliability at this level requires a co-engineering model with NVIDIA that goes beyond a traditional vendor dynamic. Teams are deeply embedded at our data centers, sharing real-time telemetry and running joint triage sessions to decompose every system halt by its root cause. This attribution-driven accountability model ensures both sides are focused on targeted improvements. While Meta optimizes scheduling and infrastructure, NVIDIA addresses firmware, signal diagnostics, and memory handling. This transparency eliminates ambiguity and drives faster fixes across both organizations. The 72-GPU Failure Domain The transition from the H100 generation’s NVL8 architecture to NVL72 has fundamentally redefined our approach to infrastructure resilience. In the NVL8 era, the scale-up domain — the group of GPUs tightly coupled via NVLink for tensor parallelism — was limited to eight units. With NVL72, that domain expands to 72 GPUs, increasing the fault blast radius by 9X. The larger scale-up domain is what gives NVL72 its bandwidth advantage, but it also makes the 72-GPU rack the unit of allocation we plan capacity around. Training parallelism maps most efficiently onto power-of-two group sizes, so each NVL72 domain runs a 64-GPU slice and keeps its remaining eight GPUs as in-rack spares. When a GPU fails, a spare takes its place immediately, so the job restarts from its last checkpoint instead of waiting on hardware repair, and the rack stays in the job. The in-rack redundancy pushes the effective rack fault rate down to the floor set by failures that can remove an entire pod; for rarer whole-rack faults we hold roughly 2% additional racks in reserve. For a 32,768 active-GPU training run, this means provisioning 523 NVL72 racks, or 37,656 GPUs, about 13% above the active footprint. Most of that margin (11%) falls out of the 64-of-72 slice geometry rather than being an absolute reliability tax. Physical complexity further intensifies these reliability requirements. A single NVL72 domain incorporates 5,184 tuned differential-pair signaling links operating at 200 Gbps, each traversing multiple potential points of failure, including GPU SerDes, compute tray PCBs, cable cartridges, and switch tray interfaces. Maintaining consistent signal integrity across this dense fabric — at 200 Gbps per lane — represents an unprecedented engineering challenge at this scale. Fault Recovery Fault recovery addresses how we react to faults to minimize
document fetched 2026-10-24 11:20 · canonical https://engineering.fb.com/2026/10/19/data-center-engineering/inside-prometheus-meta-gigawatt-scale-supercluster