loading
loading
The scale of AI data centers is typically described in terms of three resources: cost, number of GPUs (i.e., cluster size), and the amount of power required. That's because those resources are key constraints that determine how much AI processing a data center will be able to perform. But there’s another constraint that has quietly become just as critical: the back-end network that connects GPUs. The network, together with power, increasingly determines not only how many accelerators (GPUs, XPUs and other processors used for AI workloads) can be deployed in a data center, but also how efficiently and productively they can be used. Network architecture can also significantly affect how complicated the facility is to build and operate, how rapidly it can be brought online and therefore how quickly it can begin generating revenue. A network that struggles to keep pace with the demands of AI creates a hard limit on the scale and productivity of the infrastructure that can be built — what is often referred to as the network wall. For that reason, the network needs to be a primary design consideration when building and operating AI data centers. The critical role of the network The network has a direct impact on not only the design parameters of an AI data center — density, power, cost — but also on the key metrics that drive the productivity and success of the data center. The impact is so significant that a hyperscaler I spoke with recently stated that they start with the network and design their data center buildings based on what the network can support. AI processing relies on rapidly exchanging large amounts of data among accelerators synchronously and repeatedly — gradients, context, KV caches, and attention tensors are just a few examples of the data that accelerators need to exchange in coordinated operations, known as collective communication, during AI processing. Collectives such as All-to-All require every GPU across a cluster to exchange data with every other GPU, while operations such as All-Reduce and All-Gather depend on data from every other GPU in the cluster. Because these collectives are synchronized operations, all participating GPUs need to wait for all data to arrive before moving on to the next step in processing. The design and capabilities of the network have a critical impact on the performance of those operations. Network architecture decisions, topology, partitioning, oversubscription ratio, number of rails and planes, as well as technology choices such as interconnect technology, switch bandwidth and switch radix (the number of ports that can connect to a single non-blocking switch chip) determine the speed of communication among GPUs. As a result, network congestion and latency degrade the performance of AI processing. In addition to performance, the network infrastructure also impacts cost, complexity and reliability. Network design and technology determine the total number of switches, switching tiers, fibers, and optical transceivers required to create the network fabric connecting accelerators. That, in turn, impacts the number of network racks that need to be powered and cooled, the number of components that can fail, and the share of the overall budget that needs to go to the network. Those impacts ultimately drive the performance, efficiency, and scalability that determine the productivity and competitive advantage that an AI data center delivers for training and inference. Whether for an AI lab racing to develop the best model or for a neocloud competing to deliver the best inference performance for AI applications, those metrics are critical to success. One example: tokens per second, a measure of AI processing throughput, suffers when GPUs are forced to sit idle and underutilized because they are waiting for data movement across the network. In fact, the network is a major reason that GPUs in AI clusters can spend a large share of their time idle and underutilized, often below 50 percent in published production traces. Poor GPU utilization increases the time it takes to train models and the rate at which inference requests can be served. Tokens per watt, an increasingly important efficiency metric due to the challenges of securing the power necessary for AI data centers, is also directly impacted by the network. Underutilized GPUs still consume significant amounts of power, even while waiting for data. Adding network tiers, switches, and optics to support larger and larger AI data centers increases the power consumed by network switches and optics, further reducing tokens per watt. Tokens per dollar is also directly influenced by the network. Both the direct cost of the network — switches, optics, cables, etc. — and the indirect cost of underutilized GPUs contribute to the cost efficiency of an AI data center. Jensen Huang, CEO of Nvidia, highlighted that significance in his keynote address at the GTC conference in Washington, D.C., when he noted that for a 1GW data center, which costs on the order of $50 billion to build, a ten percent change in GPU utilization has an economic impact of $5bn, while SemiAnalysis indicates that the network is often the second-largest cost in an AI cluster after the accelerators themselves. Finally, the network contributes to the complexity of the data center, which impacts time to value. Every additional switch and fiber must be procured, installed, inspected, tested, configured, and monitored. More components create more opportunities for a failed optic, contaminated connector, cabling error, or underperforming link to delay completion. Taken together, that complexity can directly add to the time to bring a new data center into operation. Because of financing and lease obligations, as well as the approach of new generations of GPUs, those delays matter: a recent Carnegie Endowment financial model estimated that a one-year delay reduces the lifecycle value of an illustrative 100MW US data center by roughly $550 million, or 5.5 percent. The growing pressure on the network Despite how critical they are to the data center, AI networks are under increasing strain and struggling to keep up, not only because demand for AI training and inference is necessitating ever-larger clusters and data centers, but also because AI algorithms and architectures are evolving and advancing along several dimensions at once. GPU performance has continued to race ahead with each new generation, able to consume and generate tokens and data at higher and higher rates. For example, Nvidia's Blackwell Ultra GPU delivers roughly a 2.5x increase in compute performance over the Hopper-generation H200 that came to market only a year before. The evolution of AI models is adding additional demands on the network. Mixture-of-experts models require rapidly routing data among different experts. Longer model context windows and multimodal processing increase the size of the data set that needs to be distributed for models to process. Disaggregating prefill and decode steps in inference processing creates new data movement demands between GPUs and other accelerators. Agentic AI workloads, often orchestrated by a separate CPU rack, can trigger hundreds of concurrent API calls for inference processing during their work. The network is falling further and further behind. Network switch bandwidth and radix have been increasing at a much slower pace, with the just-emerging 102.4T generation of switches coming to market almost four years after the previous-generation 51.2T switches, continuing a historical pace of doubling only every two to four years. When bandwidth and radix per switch are insufficient, the network design is forced to expand to compensate. Additional switches and network tiers are added, which in turn require more inter-switch connections, more optical transceivers, and more cabling. Not only does that add more cost, power, and complexity; it also increases latency and congestion, leading to underperforming networks that reduce GPU utilization and slow down AI training and inference. Breaking through the network wall The network can no longer be an afterthought, considered only after determining GPUs, racks, power, and cooling architecture. By the time a bottleneck becomes visible in production, the opportunity to fix it without massive disruption has already closed. Networking architecture and technology need to be part of data center design from early stages. Flatter networks with fewer tiers, fewer switches, fewer inter-switch links, fewer optics and cables, lower power, faster commissioning, and more predictable workload performance are crucial goals that need to be part of the design. Achieving those goals requires network switches with significantly higher radix and bandwidth to keep up with the pace of computational throughput, bandwidth per GPU, larger clusters, and new AI algorithms and architectures that demand more data movement. High-radix switch platforms can deliver that while simplifying the network. That not only prevents delays in bringing a new data center online, but also improves the throughput and resulting economic impact of the entire data center over its lifetime. The gap between architectural ambition and network reality is where competitive position will be decided. Treating networking as a critical part of data center design and embracing high-radix switching will provide a crucial advantage to the winners of this race.
document fetched 2026-10-31 02:54 · canonical https://datacenterdynamics.com/en/opinions/the-network-wall-why-switches-arent-scaling-fast-enough-for-ai-data-centers