loading
loading
AI/ML Meta's Prometheus using object storage for massive AI training jobs Meta has started talking about its Prometheus gigawatt-scaleAI training cluster and its storage system has migrated from file-based to multi-tiered cached object storage to reduce GPU idle time. Meta’s Prometheus is a gigawatt-scale GPU supercluster located in New Albany, Ohio, used to build Meta’s Muse family of foundational AI models. It is being scaled up from its current hundreds of thousands of GPUs to to around 1 million GPUs functioning in lockstep to train Meta Supercluster Labs’ next frontier models. Currently there is an array of hundreds of thousands of GPUs across multiple interconnected buildings. There are two Nvidia GPU platforms within Prometheus: Catalina, which uses the Ariel variant of Nvidia’s GB200 NVL72 (Blackwell); a two-rack configuration in which 36 GPUs per rack are interconnected via NVLink to form a unified 72-GPU domain. Clemente, which uses Nvidia’s GB300 NVL72 (Blackwell Ultra) and improves on Catalina at both the GPU and the platform level. GB300 raises HBM capacity per GPU by 1.5x, from 192 GB to 288 GB, and dense FP4 throughput by a similar factor, while roughly doubling attention (softmax) acceleration. A full 72-GPU domain is deployed in a single rack. A third Nvidia GPU platform is due, San Miguel, based on Vera Rubin NVL72 GPUs. This will be will be a generational leap in per-GPU performance and will integrate directly into Prometheus’s existing network fabric. As the cluster matures, it will deliver even more compute per watt by augmenting its capacity (primarily interconnected GB300s) with Vera Rubin. Meta is building a second supercluster in Richmond Parish, Louisiana, called Hyperion. There are several Prometheus datacenter buildings at New Albany. Inside a building an Ethernet-based fabric is used. Prometheus cluster buildings are interconnected by a multi-layer hub-and-spoke Backend Aggregation (BAG) network that spans from 2 km to over 100 km. This uses Arista 7800-series chassis running deep-buffered Broadcom Jericho3 silicon, delivering multiple Pbps of bandwidth per region pair. Prometheus Storage Scheme Outline This is a custom, multi-tier storage stack built on Meta’s Tectonic foundational layer, with BLOB-storage (Binary Large OBject) interfaces and specialized hardware platforms for bulk and high-performance needs. The storage subsystem features a 4-layer hardware stack: - L1/L2: Memory and local flash on GPU hosts. - L3: Regional flash-backed BLOB storage. - L4: Source of truth: durable HDD-backed BLOB/Tectonic storage (global or regional). There is distributed caching (leveraging spare GPU-host memory, reusing components from Meta’s Owl system), metadata caching, prefetching in the data loader, hedged reads, and dynamic concurrency control. The design prioritizes predictable low tail latencies so training steps are not stalled by storage. Owl Owl is Meta’s production system for moving massive volumes of hot internal content efficiently by combining peer-to-peer data transfer with intelligent centralized control. It is a high-fanout (from a few clients to millions of processes) content distribution system for large, frequently accessed (“hot”) objects inside Meta’s private cloud, delivering executables/binaries, code artifacts, AI models, and search indexes to millions of hosts. The delivered content can range from a few GMB to several TB in size. There are separate data and control planes: - Decentralized data plane: Ephemeral peer-to-peer distribution trees. Data streams from sources through peers as soon as chunks arrive (avoiding full store-and-forward delays). Trees are short-lived and per-chunk. - Centralized control plane: Tracker services maintain detailed, up-to-date metadata on peers, their cache contents, locations, and ongoing downloads. Trackers decide sources, retry behavior, caching/eviction, and tree construction. Prometheus Storage Software Stack The Prometheus storage stack has migrated away from its Llama-training design of having an exposing an NFS-like FileSystem interface on top of Tectonic (see bootnote). This architecture continues to be used widely within Meta, but Prometheus is moving to BLOB-based storage. This “is motivated by the need for unified storage access to massive data lakes in the BLOB-storage layer as well as the need for high performance” to maximize GPU utilization. The storage software stack has two main layers: - Tectonic - the foundational, regional, multi-tenant block storage fabric. It provides high durability/availability via erasure coding, media tiering (HDD + flash), and smart placement of hot/warm/cold data. - BLOB-storage layers on top of Tectonic: These expose object-storage (and historically file-system) APIs. There is a unified flat metadata schema backed by ZippyDB (for fast O(1) path-to-address lookups), elimination of a dataplane proxy in favor of a “fat client” SDK that streams data directly from storage servers, and regional deployments colocated with GPUs. Meta now deploys a regional BLOB-storage stack colocated with GPUs in every AI region. When the SDK on the client receives this API call, it now issues a getReadPlan(“/bucket/path”) request to the API server. The API server does O(1) lookup per chunk to the new metadata store to map the path to (blockId, offset, size) tuples. It then returns the ReadPlanResult to the SDK. The SDK has Tectonic BlockClient embedded within it, and so is now able to stream data from these blocks directly from Tectonic. Two caches are used: - Distributed data cache: Prometheus leverages the spare memory on the GPU hosts as a distributed data cache for frequently and concurrently accessed data. To achieve this it reused components from Meta’s Owl subsystem: integrating the peers in the Owl subsystem directly into the BLOB-storage client SDK so that all data access goes through this data cache. - Readplan metadata cache: Readplan refers to the mapping from path to storage address. It caches the read-plan for frequently accessed BLOBs in a distributed-memory store similar to memcache. Prometheus tiered caching We can now flesh out the storage scheme outline. Meta says the core idea is to have a dataloading scheme leveraging the memory and flash on the GPU host as L1 and L2 caches, and regional BLOB-storage fabric backed by flash as the L3 cache. The global BLOB-storage HDD-based data lake is the backing store, what Meta calls “the ultimate source of truth." A dataloader accesses the caches and this cache-backing store system through the BLOB-storage SDK. Note that: - Dataloader prefetch: Dataloaders prefetch the next batch of datasets into memory while processing the current batch. This prefetch will surface as a read operation at the BLOB-storage SDK level. - Deep prefetch: We expose an explicit prefetch() API as part of the BLOB-storage SDK. The dataloader will trigger explicit prefetch of the data needed during the next few minutes by invoking the prefetch() API in the background. This API triggers hydration of data from remote storage onto the local region L3 cache and also prewarms the metadata cache. - Automatic data life cycle: Data in the L3 regional disaggregated flash tier is typically held for a configured period of time to allow reuse across epochs in a training cycle. We support custom eviction policies, including TTL and LRU policies. The eviction policies are also capacity/quota aware. Once Meta stared using this dataloading scheme, average job ingestion time dropped from 150 minutes to 10 minutes. The maximum very large job ingestion time of 89 hours fell to just over 3 hours; 182 minutes. Storage hardware The storage hardware includes: - Olympic and Yosemite V3.5 Crater Lake: Flash/SSD-based platforms optimized for lower-latency, higher-throughput access. - Grand Canyon: HDD-based system for bulk training-data storage. - Support for general-purpose compute platforms (e.g., Yosemite variants) for cluster management. These allow independent optimization of capacity, latency, and cost across the data pipeline Yosemite v3.5 has a blade/sled chassis design with up to a dozen 10 RU servers in a 40 RU rack. It supports carious compute, memory/CXL, PICe, accelerator and SSD expansions. Yosemite V3.5 uses OCP-aligned datacenter NVMe SSDs (E1.S for capacity/performance storage and M.2 for boot), not commodity consumer drives. They are purpose-built or qualified datacenter NVMe devices. Crater Lake is a server designed for the Yosemite v3.5 architecture, supporting 4th Gen Intel Xeon (Sapphire Rapids / Eagle Stream) processors. It has a a single-socket 1OU (1 Open Unit) design and is intended as a base compute node. The Open Unit is specific to the Open Compute Project’s Open Rack Standard, being 48 mm (about 1.89 inches or 1⅞ inches) high, not the traditional EIA 1U = 1.75-inch rack unit. Grand Canyon is a 4OU-tall, single-drawer dense chassis that holds up to 72 × 3.5-inch HDDs. There are three build options: - Dual storage server — Two compute nodes + 72 HDDs (all drives powered for higher performance/access speed). Fully standalone. - Single storage server — One compute node acting as a head node, which can connect to additional JBOD chassis. - Pure JBOD — No compute; used as a disk shelf. In head-node + JBOD setups this can scale to 216 drives (or more in multi-chassis configurations). It’s another design, like Yosemite and Crater Lake, that Meta contributed to the OCP. Checkpointing Prometheus uses a distributed checkpointing architecture that asynchronously writes model state (including weights, gradients, and optimizer data) to multi-tier storage. It employs delta checkpointing and in-memory staging to minimize impact on training throughput. This fits the overall tiered model (fast local/in-memory staging → flash → durable HDD-backed storage) and supports high goodput (e.g., >85% on pre-training runs) Takeaway This specialized custom storage scheme has been developed by Meta to minimize GPU waits in very large scale training clusters. If organizations are running similar scale AI training workloads, meaning they are hyperscaler or near hyperscaler class, then they could write the software themselves or, if smaller and/or the standalone object storage trio; Cloudian, MinIO and Scality, can provide the functionality they require, use that software instead. The OCP-class NVMe SSDs should be available from OCP suppliers and commodity disk drives are commodity disk drives. It will be interesting to see if Meta uses its Prometheus clusters for AI inferencing and agentic AI workloads, which won’t have, we suppose, such large job ingestion times. Neither will individual inferencing jobs require huge numbers of GPUs working in lockstep. Bootnotes In 2024 Meta said Hammerspace was its data orchestration software supplier, supporting 49,152 Nvidia H100 GPUs split into two equal clusters and used in two Llama 3 AI training systems. Meta’s Grand Teton OCP hardware chassis houses the GPUs, which rely on Meta’s Tectonic distributed, flash-optimized and exabyte scale storage system. This was accessed though a Meta-developed Linux Filesystem in Userspace (FUSE) API and used for AI model data needs and model checkpointing. Both the Tectonic and Hammerspace-backed storage deployments use Meta’s YV3 Sierra Point server fitted with the highest-capacity E1.S format SSDs available. A previous AI Research SuperCluster, with 16,000 Nvidia A100 GPUs, was used to build Meta’s gen 1 AI models and “continues to play an important role in the development of Llama and Llama 2, as well as advanced AI models for applications ranging from computer vision, NLP, and speech recognition, to image generation, and even coding.” That cluster used Pure Storage FlashArray and FlashBlade all-flash arrays. The Meta BLOB-storage SDK is the client-side library that lets applications, like AI training jobs, efficiently read and manage BLOBs from Meta’s global storage fabric, with direct streaming, Owl-based distributed caching, and AI-oriented optimizations.
AI/ML Meta's Prometheus using object storage for massive AI training jobs Meta has started talking about its Prometheus gigawatt-scaleAI training cluster and its storage system has migrated from file-based to multi-tiered cached object storage to reduce GPU idle time. Meta’s Prometheus is a gigawatt-scale GPU supercluster located in New Albany, Ohio, used to build Meta’s Muse family of foundational AI models. It is being scaled up from its current hundreds of thousands of GPUs to to around 1 million GPUs functioning in lockstep to train Meta Supercluster Labs’ next frontier models. Currently there is an array of hundreds of thousands of GPUs across multiple interconnected buildings. There are two Nvidia GPU platforms within Prometheus: Catalina, which uses the Ariel variant of Nvidia’s GB200 NVL72 (Blackwell); a two-rack configuration in which 36 GPUs per rack are interconnected via NVLink to form a unified 72-GPU domain. Clemente, which uses Nvidia’s GB300 NVL72 (Blackwell Ultra) and improves on Catalina at both the GPU and the platform level. GB300 raises HBM capacity per GPU by 1.5x, from 192 GB to 288 GB, and dense FP4 throughput by a similar factor, while roughly doubling attention (softmax) acceleration. A full 72-GPU domain is deployed in a single rack. A third Nvidia GPU platform is due, San Miguel, based on Vera Rubin NVL72 GPUs. This will be will be a generational leap in per-GPU performance and will integrate directly into Prometheus’s existing network fabric. As the cluster matures, it will deliver even more compute per watt by augmenting its capacity (primarily interconnected GB300s) with Vera Rubin. Meta is building a second supercluster in Richmond Parish, Louisiana, called Hyperion. There are several Prometheus datacenter buildings at New Albany. Inside a building an Ethernet-based fabric is used. Prometheus cluster buildings are interconnected by a multi-layer hub-and-spoke Backend Aggregation (BAG) network that spans from 2 km to over 100 km. This uses Arista 7800-series chassis running deep-buffered Broadcom Jericho3 silicon, delivering multiple Pbps of bandwidth per region pair. Prometheus Storage Scheme Outline This is a custom, multi-tier storage stack built on Meta’s Tectonic foundational layer, with BLOB-storage (Binary Large OBject) interfaces and specialized hardware platforms for bulk and high-performance needs. The storage subsystem features a 4-layer hardware stack: - L1/L2: Memory and local flash on GPU hosts. - L3: Regional flash-backed BLOB storage. - L4: Source of truth: durable HDD-backed BLOB/Tectonic storage (global or regional). There is distributed caching (leveraging spare GPU-host memory, reusing components from Meta’s Owl system), metadata caching, prefetching in the data loader, hedged reads, and dynamic concurrency control. The design prioritizes predictable low tail latencies so training steps are not stalled by storage. Owl Owl is Meta’s production system for moving massive volumes of hot internal content efficiently by combining peer-to-peer data transfer with intelligent centralized control. It is a high-fanout (from a few clients to millions of processes) content distribution system for large, frequently accessed (“hot”) objects inside Meta’s private cloud, delivering executables/binaries, code artifacts, AI models, and search indexes to millions of hosts. The delivered content can range from a few GMB to several TB in size. There are separate data and control planes: - Decentralized data plane: Ephemeral peer-to-peer distribution trees. Data streams from sources through peers as soon as chunks arrive (avoiding full store-and-forward delays). Trees are short-lived and per-chunk. - Centralized control plane: Tracker services maintain detailed, up-to-date metadata on peers, their cache contents, locations, and ongoing downloads. Trackers decide sources, retry behavior, caching/eviction, and tree construction. Prometheus Storage Software Stack The Prometheus storage stack has migrated away from its Llama-training design of having an exposing an NFS-like FileSystem interface on top of Tectonic (see bootnote). This architecture continues to be used widely within Meta, but Prometheus is moving to BLOB-based storage. This “is motivated by the need for unified storage access to massive data lakes in the BLOB-storage layer as well as the need for high performance” to maximize GPU utilization. The storage software stack has two main layers: - Tectonic - the foundational, regional, multi-tenant block storage fabric. It provides high durability/availability via erasure coding, media tiering (HDD + flash), and smart placement of hot/warm/cold data. - BLOB-storage layers on top of Tectonic: These expose object-storage (and historically file-system) APIs. There is a unified flat metadata schema backed by ZippyDB (for fast O(1) path-to-address lookups), elimination of a dataplane proxy in favor of a “fat client” SDK that streams data directly from storage servers, and regional deployments colocated with GPUs. Meta now deploys a regional BLOB-storage stack colocated with GPUs in every AI region. When the SDK on the client receives this API call, it now issues a getReadPlan(“/bucket/path”) request to the API server. The API server does O(1) lookup per chunk to the new metadata store to map the path to (blockId, offset, size) tuples. It then returns the ReadPlanResult to the SDK. The SDK has Tectonic BlockClient embedded within it, and so is now able to stream data from these blocks directly from Tectonic. Two caches are used: - Distributed data cache: Prometheus leverages the spare memory on the GPU hosts as a distributed data cache for frequently and concurrently accessed data. To achieve this it reused components from Meta’s Owl subsystem: integrating the peers in the Owl subsystem directly into the BLOB-storage client SDK so that all data access goes through this data cache. - Readplan metadata cache: Readplan refers to the mapping from path to storage address. It caches the read-plan for frequently accessed BLOBs in a distributed-memory store similar to memcache. Prometheus tiered caching We can now flesh out the storage scheme outline. Meta says the core idea is to have a dataloading scheme leveraging the memory and flash on the GPU host as L1 and L2 caches, and regional BLOB-storage fabric backed by flash as the L3 cache. The global BLOB-storage HDD-based data lake is the backing store, what Meta calls “the ultimate source of truth." A dataloader accesses the caches and this cache-backing store system through the BLOB-storage SDK. Note that: - Dataloader prefetch: Dataloaders prefetch the next batch of datasets into memory while processing the current batch. This prefetch will surface as a read operation at the BLOB-storage SDK level. - Deep prefetch: We expose an explicit prefetch() API as part of the BLOB-storage SDK. The dataloader will trigger explicit prefetch of the data needed during the next few minutes by invoking the prefetch() API in the background. This API triggers hydration of data from remote storage onto the local region L3 cache and also prewarms the metadata cache. - Automatic data life cycle: Data in the L3 regional disaggregated flash tier is typically held for a configured period of time to allow reuse across epochs in a training cycle. We support custom eviction policies, including TTL and LRU policies. The eviction policies are also capacity/quota aware. Once Meta stared using this dataloading scheme, average job ingestion time dropped from 150 minutes to 10 minutes. The maximum very large job ingestion time of 89 hours fell to just over 3 hours; 182 minutes. Storage hardware The storage hardware includes: - Olympic and Yosemite V3.5 Crater Lake: Flash/SSD-based platforms optimized for lower-latency, higher-throughput access. - Grand Canyon: HDD-based system for bulk training-data storage. - Support for general-purpose compute platforms (e.g., Yosemite variants) for cluster management. These allow independent optimization of capacity, latency, and cost across the data pipeline Yosemite v3.5 has a blade/sled chassis design with up to a dozen 10 RU servers in a 40 RU rack. It supports carious compute, memory/CXL, PICe, accelerator and SSD expansions. Yosemite V3.5 uses OCP-aligned datacenter NVMe SSDs (E1.S for capacity/performance storage and M.2 for boot), not commodity consumer drives. They are purpose-built or qualified datacenter NVMe devices. Crater Lake is a server designed for the Yosemite v3.5 architecture, supporting 4th Gen Intel Xeon (Sapphire Rapids / Eagle Stream) processors. It has a a single-socket 1OU (1 Open Unit) design and is intended as a base compute node. The Open Unit is specific to the Open Compute Project’s Open Rack Standard, being 48 mm (about 1.89 inches or 1⅞ inches) high, not the traditional EIA 1U = 1.75-inch rack unit. Grand Canyon is a 4OU-tall, single-drawer dense chassis that holds up to 72 × 3.5-inch HDDs. There are three build options: - Dual storage server — Two compute nodes + 72 HDDs (all drives powered for higher performance/access speed). Fully standalone. - Single storage server — One compute node acting as a head node, which can connect to additional JBOD chassis. - Pure JBOD — No compute; used as a disk shelf. In head-node + JBOD setups this can scale to 216 drives (or more in multi-chassis configurations). It’s another design, like Yosemite and Crater Lake, that Meta contributed to the OCP. Checkpointing Prometheus uses a distributed checkpointing architecture that asynchronously writes model state (including weights, gradients, and optimizer data) to multi-tier storage. It employs delta checkpointing and in-memory staging to minimize impact on training throughput. This fits the overall tiered model (fast local/in-memory staging → flash → durable HDD-backed storage) and supports high goodput (e.g., >85% on pre-training runs) Takeaway This specialized custom storage scheme has been developed by Meta to minimize GPU waits in very large scale training clusters. If organizations are running similar scale AI training workloads, meaning they are hyperscaler or near hyperscaler class, then they could write the software themselves or, if smaller and/or the standalone object storage trio; Cloudian, MinIO and Scality, can provide the functionality they require, use that software instead. The OCP-class NVMe SSDs should be available from OCP suppliers and commodity disk drives are commodity disk drives. It will be interesting to see if Meta uses its Prometheus clusters for AI inferencing and agentic AI workloads, which won’t have, we suppose, such large job ingestion times. Neither will individual inferencing jobs require huge numbers of GPUs working in lockstep. Bootnotes In 2024 Meta said Hammerspace was its data orchestration software supplier, supporting 49,152 Nvidia H100 GPUs split into two equal clusters and used in two Llama 3 AI training systems. Meta’s Grand Teton OCP hardware chassis houses the GPUs, which rely on Meta’s Tectonic distributed, flash-optimized and exabyte scale storage system. This was accessed though a Meta-developed Linux Filesystem in Userspace (FUSE) API and used for AI model data needs and model checkpointing. Both the Tectonic and Hammerspace-backed storage deployments use Meta’s YV3 Sierra Point server fitted with the highest-capacity E1.S format SSDs available. A previous AI Research SuperCluster, with 16,000 Nvidia A100 GPUs, was used to build Meta’s gen 1 AI models and “continues to play an important role in the development of Llama and Llama 2, as well as advanced AI models for applications ranging from computer vision, NLP, and speech recognition, to image generation, and even coding.” That cluster used Pure Storage FlashArray and FlashBlade all-flash arrays. The Meta BLOB-storage SDK is the client-side library that lets applications, like AI training jobs, efficiently read and manage BLOBs from Meta’s global storage fabric, with direct strea
document fetched 2026-10-24 11:21 · canonical https://blocksandfiles.com/ai-ml/2026/10/20/metas-prometheus-using-object-storage-for-massive-ai-training-jobs/5304117