The Breaking Points: Networking Strains Under AI’s Scale Demands

extract

The Breaking Points: Networking Strains Under AI’s Scale Demands As AI moves from pilots to production, synchronized traffic, microbursts, and east–west patterns are pushing legacy architectures, tooling, and operations to their limits. In the first of a three-part series examining where AI infrastructure struggles to scale, we take a closer look at the networking challenges reshaping enterprise systems. Imagine a network engineer starting the day with a fresh cup of coffee, troubleshooting an AI deployment where everything appears to be working. The GPUs have headroom, the models are running, and the servers are humming. Suddenly, performance begins to degrade – responses slow, delays compound, and systems fall out of sync. The engineer spills her coffee as the issue becomes clear: The culprit isn’t compute. It’s the network, struggling to keep up with the volume and variability of traffic moving between systems. As enterprises scale AI workloads, this scenario is becoming harder to ignore. While much of the industry focus has centered on GPUs and model performance, the fabric connecting those resources is emerging as an early point of strain as organizations move beyond pilot deployments. AI is also rapidly increasing pressure on infrastructure. Global data center capacity is projected to nearly double to around 200 GW by 2030, according to JLL. “Less than 10% of US inventory is capable of true AI-dense critical load today,” Sean Farney, vice president of data center strategy at JLL, told Data Center Knowledge. At the same time, nearly half of the planned US data center projects for 2026 are expected to face delays or cancellations, according to Bloomberg reporting based on project pipeline data. The result is a growing mismatch between how AI operates and how enterprise networks are designed. From Design Limits to Real-World Strain Traditional enterprise networks were built for predictable traffic patterns, scheduled workloads, and relatively stable application environments. AI changes those assumptions. Inference workloads generate bursts of traffic, while agent-driven systems introduce constant back-and-forth communication between services. Distributed architectures push data across clouds, data centers, and edge environments in real time. This combination is exposing limits in latency control and routing at scale. “In the data center, people are finding it difficult to integrate the new networks they build out for AI with legacy infrastructure,” said Shamus McGillicuddy, vice president of research at Enterprise Management Associates. “This might include protocol mismatches – network borders where RoCE collides with TCP/IP.” On the WAN side, he said, organizations are encountering new complexity around edge inference as they distribute workloads across sites while managing latency. Network demand is already shifting. Sameh Boujelbene, vice president at Dell’Oro Group, said AI backend networks accounted for roughly one-third of the Ethernet data center switch market in 2025 and drove about 80% of year-over-year revenue growth. That demand is accelerating the transition to higher-speed interconnects. Boujelbene said 800 Gbps adoption is unfolding far faster than previous generations, reaching tens of millions of port shipments within three years, far outpacing the ramp of 400 Gbps. Kevin Wollenweber, senior vice president and general manager of Data Center and Internet Infrastructure at Cisco, said the shift from training clusters to inference and agent-driven workloads is already changing network behavior. “These systems are going to be operating continuously,” Wollenweber said. “It’s not just bursty traffic anymore – it’s sustained, 24/7 demand on the network.” Synchronization Pressure and Microbursts As AI systems scale, performance is increasingly shaped by how well networks handle synchronization across large clusters. Training workloads require coordination across thousands of GPUs, where each compute cycle depends on all nodes completing communication steps before the next begins. “When you have thousands of GPUs finishing a compute cycle and simultaneously sending data, you get a microburst effect that can overwhelm switch buffers,” said Ron Westfall, vice president and analyst at HyperFrame Research. That can lead to dropped packets and retransmissions, introducing delays that ripple across the system. In these environments, performance is dictated not by average latency, but by tail latency – the slowest operation in the system. In synchronized training workloads, a single delayed packet can stall progress across the entire cluster. Wollenweber said many enterprise environments are not yet prepared for the scale of AI-driven traffic. “I think a lot of enterprises just aren’t ready for the sheer capacity that’s going to hit them,” he said, pointing to the jump from gigabit-scale connectivity to terabit-level throughput in AI clusters. Westfall noted that the challenge is not simply bandwidth, but coordination at scale, as communication overhead increases across distributed workloads. Visibility Fails First For many operators, the bigger problem is not that networks are failing outright – but that failures are difficult to see. Performance degradation often occurs without clear signals such as link failures or outages, leaving teams with limited insight into what is actually happening inside the network. Westfall described these as “gray failures,” where systems appear healthy but performance is inconsistent or degraded. “Just scaling past thousands of GPUs can become an exercise in phantom troubleshooting,” he said, as teams struggle to identify the root cause of issues that do not trigger traditional alerts. Part of the challenge is visibility into the transport layer. Without detailed insight into how traffic is flowing – particularly in RDMA-based environments – operators can miss congestion events and microbursts that impact performance. Boujelbene said AI traffic patterns are fundamentally different from traditional enterprise workloads, dominated by large, long-lived “elephant flows,” and highly synchronized many-to-many communication. These patterns are extremely sensitive to latency and jitter, requiring high bandwidth and near-zero packet loss. In AI environments, where traffic is predominantly east-west rather than north-south, traditional network designs and monitoring approaches can obscure where problems originate. Gray failures degrade performance without tripping traditional alerts, complicating root-cause analysis. (Image: Alamy) Manual Operations Struggle to Keep Up Network operations teams are also hitting limits. Many still rely on manual configuration, ticket-driven workflows, and fixed maintenance windows – approaches that are difficult to reconcile with AI systems that operate continuously and adapt in real time. “Current NetOps models are not ready for AI networks,” McGillicuddy said. “They need to automate more, and alert noise is out of control.” He added that traditional monitoring approaches, such as SNMP polling, often miss microbursts and short-lived congestion events, creating blind spots in performance visibility. “There is growing demand for real-time streaming telemetry,” he said, along with packet-level analytics capable of identifying AI-driven traffic patterns. Wollenweber said the challenge is not just capacity, but coordination across systems. “The tools that were built to manage traditional data center fabrics aren’t built for managing GPU workloads and large AI clusters,” he said. An Early Signal of Broader Strain The pressure on networking may be an early indicator of broader infrastructure challenges. As AI workloads scale, similar stress points are likely to emerge across other layers, including power, cooling, and operations. The challenge goes beyond compute and encompasses the entire surrounding system. Boujelbene said the network is increasingly becoming a limiting factor in AI deployments, as performance depends on keeping large clusters fully utilized. “The network is no longer just plumbing – it’s a first-order limiter of AI performance and scaling efficiency,” she said. The visibility gap is unlikely to close quickly. Westfall said many organizations are still early in adopting the tooling needed to monitor AI traffic at scale, and that closing that gap will take time as new observability and automation capabilities are deployed. As AI workloads continue to scale, networking is emerging as a critical bottleneck, exposing broader challenges across infrastructure layers. Addressing these issues will require a combination of real-time visibility, automation, and architectural innovation to ensure AI systems can operate efficiently at scale. In the next installment, we’ll explore how power and cooling systems are adapting to AI’s scale demands.