The Switch Is the Bottleneck: Why AI Infrastructure Has a Network Problem
retained text
The Switch Is the Bottleneck: Why AI Infrastructure Has a Network Problem AI workloads are outpacing network capabilities, leaving expensive chips idle. Mark Rushworth explains why the switch is the bottleneck and how to fix it. July 2, 2026 The rapid growth of AI workloads is exposing a critical bottleneck in infrastructure: the network. While compute power has surged, the ability of networks to keep pace has not. This imbalance is leaving some of the world’s most advanced chips underutilized, driving up costs and energy consumption. A study of Model Floating Point Operations Per Second (FLOPs) Utilization – a key indicator of computational efficiency – shows AI labs achieving just 35-40% Model Flop Utilization (MFU) on Nvidia H100S during trillion-parameter training runs, meaning the world’s most expensive chips are idle for more than half the time as they wait for data to arrive over the network. The network fabric connecting compute is now the binding constraint on what AI systems can actually do. The architectural decisions being made now, stitching together components designed in isolation, will determine the cost, energy efficiency, and competitive viability of future AI infrastructure. Bandwidth Is Surging While the Network Can’t Keep Pace AI training workloads are already moving beyond 400 Gb/s to 800 Gb/s, with 1.6 Tb/s line rates on the near-term roadmap, but raw link speed is only part of the problem. As clusters scale to thousands of GPUs, the challenge shifts from raw connection speed to how efficiently the switching fabric coordinates data movement across all of them. This is a fundamentally harder engineering problem. Network technology must achieve the 1.6 Tb/s line rate by 2027. Miss that window will force the ecosystem to route around you. This is why networking’s share of data center capex is rising from roughly 5-10% today toward 15-20% by 2030. Networking is now a primary cost driver, not an infrastructure overhead. Why the Bottleneck Can’t Be Fixed With Interconnects Alone The instinctive response – faster transceivers, denser cables, and higher line rates – does not solve the underlying problem.. As per-link bandwidth increases, the demands placed on every switching node in the network grow. A switch that was marginal at 400 Gb/s becomes a hard ceiling at 800 Gb/s. The interconnect upgrade exposes the switching layer, weighing down the entire process. The consequences of attempting to bypass this bottleneck are concrete. Building cluster scale purely through point-to-point interconnects, routing around the switching layer to avoid its limitations, multiplies the laser sources required, drives power consumption up nonlinearly, and compounds complexity with every node added. The switch is unavoidable. The only question is whether it performs well enough to no longer be the bottleneck. A Disaggregated Industry Building Integrated Systems The AI infrastructure stack evolved as a collection of independently optimized parts of accelerators, transceivers, interconnects, switches, each developed to its own performance envelope, then handed to architects expected to make them work together. The result is overengineering and wasted capacity. Designers must assume worst-case conditions at every interface. Network fabrics are specified for a generic workload that matches no actual deployment. This is the crux of the issue: the industry is trying to solve a system-level problem with component-level thinking. Wasted switching capacity leads to underutilized compute and power budgets that, by design, exceed actual needs. Trillion-dollar investments end up substantially less productive than they should be. Fixing the Bottleneck in Practice Closing the gap between raw compute and delivered performance demands a different starting point. Rather than assembling a fabric from the best available parts, AI network architecture must begin with the workload and reason backward to the switching, interconnect, and interposer design that actually serves it. In practice, this means three things. Co-optimization across the stack: the interposer, interconnect, and switching layer are not independent variables. The network's performance envelope is set by how these layers interact: gains in one are routinely offset by constraints in another. Architecture-specific design: AI training, AI inference, and HPC workloads have fundamentally different traffic patterns, latency tolerances, and bandwidth utilization profiles. A reference architecture for training looks materially different from one designed for inference; generic designs serve neither workload well. Photonic packet-level, reconfigurable switching: Electronic packet switches hit hard limits at scale: power dissipation grows, latency floors don’t move, and silicon hits physical constraints. Photonic switching offers a path through these limits, but the architecture of that switching matters as much as the medium. Circuit switching suits predictable, long-duration flows. AI traffic is asymmetric, shifting dynamically as workloads move between training and inference. Photonic circuit switching cannot reconfigure quickly enough to avoid the idle periods that defeat the purpose of optical systems. Packet-level, reconfigurable photonic architectures solve this, preserving the low latency, high bandwidth, and energy efficiency of the optical medium. The Future of Networking Nvidia has made networking one of its biggest divisions for good reason: returns on compute depend on whether the network can deliver at the speed required, without latency, congestion, or underutilization. The switching layer must be a first-class design input. Existing switching architectures were built for a different era, and AI traffic patterns have already outgrown them. The winners will be those who design from the workload outward, adopting architectures that can match the nature of AI traffic as it continues to grow. The rest will continue paying for compute they can’t use.
document fetched 2026-07-03 09:03 · canonical https://datacenterknowledge.com/switches-routers/the-switch-is-the-bottleneck-why-ai-infrastructure-has-a-network-problem