How to Evaluate CPU Performance for Agentic AI

extract

SPONSORED FEATURE How to Evaluate CPU Performance for Agentic AI A funny thing happened on the way from machine learning to chatbots to agentic AI: the central processing unit, or CPU, that has dominated commercial computing for six decades and that was temporarily supplanted by GPUs and XPUs that are massively parallel and therefore very good at crunching lots of vector and matrix math, became central again. So much so that the CPU market is expected to grow from $26 billion in 2025 to $246 billion only five years later, with the bulk of that growth coming from AI host CPUs and agentic AI sandboxes for executing work as part of an agentic AI workflow. But here’s the rub: not all CPUs are created equal, and hyperscalers, cloud builders, and AI model builders as well as neoclouds, sovereign AI centers, and enterprises are going to have to balance a desire to restrict the number of different CPU SKUs in their fleet against an equally strong desire to get a perfect match between different parts of the enterprise workflow that now includes agentic AI and the underlying hardware that supports all of those distinct pieces of human-written and AI-generated code that embodies their workflows. And let’s not forget that the inference that underpins agentic AI is itself changing very fast. This makes shopping for CPUs more complex than we have seen in the past with ever more complex workloads in the datacenter. At datacenter scale, the question becomes how many successful tasks can be completed within a rack’s power and thermal envelope. That is why Arm is advocating that customers must examine the performance of CPUs at the socket, the system, and the rack level, with an eye on both single thread performance of a CPU core and the scaling of cores within a socket, across a system, and spanning a rack to aggregate performance that can be used to run many AI agents concurrently while meeting the SLAs. This is not the first time that the IT industry has had to deal with complex multi-tier, multi-system workloads and tried to create benchmark tests to simulate that complexity so companies could make intelligent purchasing decisions. The TPC-E benchmark, for instance, simulated the workings of a warehouse – the real kind, with forklifts, not a data warehouse – and had a mix of interactive and batch workloads running across database, application, and infrastructure tiers. “Multi-tier and stateful computing are not new, and existing cloud and enterprise benchmarks remain important for understanding individual parts of the system,” Jeff Defilippi, Senior Director, Product Management at Arm, tells The Next Platform. “What changes with agentic AI is how those components are composed. A single task can dynamically chain model calls, retrieval, database transactions, code execution, tool invocations and state changes, with the result of one step determining what happens next. That makes successful task completion, not any single component metric, the more meaningful unit of performance.” “So that does not make existing benchmarks less useful. We still need component-level measurements to understand where performance differences come from,” Defilippi continues. “But they need to be connected to end-to-end measurements that show how quickly and efficiently the system completes the task, and how that performance holds up as more agents run concurrently and the overall server utilization increases.” By the way, Arm is not advocating for some kind of golden agentic AI benchmark. Arm is instead advocating for a methodology that combines established component benchmarks with agentic workload measurements and end-to-end task evaluation. Benchmarks such as InferenceX and TPCx-AI can represent important portions of the workflow, including inference serving and the machine learning pipeline. But an agentic system also includes CPU-side orchestration, retrieval, state management, tool execution, sandboxing, security, and other activities that those discrete benchmarks were not designed to test. “The goal is therefore to take a holistic view of the agentic flow while retaining component-level measurements that let engineers understand where performance differences originate,” says Defilippi. “Arm intends to characterize a broad set of these behaviors, from targeted component tests through higher-level metrics such as wall-clock task completion, agent density, and efficiency. The methodology is applicable to AGI CPU systems, but we are defining the workload and measurement framework rather than announcing a specific AGI CPU cluster benchmark.” The methodology that Arm is proposing will not be restricted to CPUs, of course, because agentic AI doesn’t only run on CPUs. Component-level performance of not only CPUs, but also GPUs or other XPU accelerators, memory subsystems, and networking has to be measured, and we also have to pay close attention to how orchestration, retrieval, and tool deployment and execution all run. A useful agentic AI testing methodology needs both controlled component tests that expose individual bottlenecks and end-to-end tests that measure user outcomes. When it comes to agentic AI, sequential latency – the bailiwick of the CPU, which is a moderately parallel compute engine that has cache hierarchies and clock speeds optimized to have high sequential processing capability – is a key constraint for task completion in an agent loop. The CPUs in the agentic AI infrastructure have to wait for GPUs or a mix of GPUs and XPUs to do the prefill token processing and the decode token output that drives the agentic workflow, spawning queries and activation of tools and often the generation of code. “Single-threaded performance matters because many parts of an agent loop are latency-sensitive and sit on the critical path,” Defilippi says. “But it is only one dimension. Agent infrastructure also has to support many agents at once, so core count, memory bandwidth, cache capacity, scheduling efficiency and sustained performance under high utilization become increasingly important as concurrency rises.” Obviously, benchmark measurement problems emerge when a single logical operation spans multiple dependent steps across the system. An agent may reason about a task, invoke a tool, wait for a database or API response, update its state, return to model inference and then decide what to do next. Because later steps often depend on the results of earlier ones, latency accumulates across the critical path even when different parts of the underlying system can operate in parallel. “Many agentic workflows therefore have both a latency problem and a concurrency problem,” explains Defilippi . “For an agent, the next action often cannot begin until the preceding dependency completes, so CPU-side execution, memory access, model inference, tool calls and network latency can all contribute to task-completion time. Reducing latency in the CPU-visible portions of that path can move the agent to its next action sooner and, where accelerators are involved, return work to the GPU or XPU sooner as well.” “At the same time, the infrastructure is rarely running just one agent. Other agents and independent portions of a workflow can execute while one dependency is waiting. As concurrency increases, core count, memory bandwidth, cache capacity, scheduling efficiency and sustained performance under utilization become increasingly important. The system therefore has to balance responsive execution along each dependency chain with enough aggregate throughput to support many agents concurrently.” That leads to a practical way to evaluate CPU performance at three levels. At the core level, measure latency and per-thread performance for work that sits on the critical path. At the system level, measure throughput, concurrency, memory behavior and sustained performance under high utilization. At the rack level, measure how much completed work the infrastructure can sustain within its power, cooling and physical constraints. Component measurements explain why one platform behaves differently from another. End-to-end agentic measurements show whether those differences actually improve successful task completion. In systems where accelerator inference is part of the agent loop, CPU performance can also affect how efficiently those accelerators are used. The CPU handles work such as request scheduling, orchestration, data preparation, runtime services and other functions around model execution. Delays on the CPU side can extend the overall task path, while efficient CPU execution can return work to the accelerator sooner. Infrastructure teams should therefore evaluate performance under realistic load and concurrency rather than relying only on lightly loaded peak results. As has always been the case, the architecture of a system has to reflect the workload, and different choices for compute engines, memory, networking, and storage will be made based on the nature of those workloads. And, to make matters even more complex, it is very difficult to predict how the agentic AI workloads will evolve over the next few years. But there are considerations between latency and concurrency that will persist no matter how agentic AI architecture changes over the next four to five years – a span of time that matches how long machines installed in the field today will still be running. Defilippi explains this interplay: “Where inference serving is part of the agent loop, CPU-visible functions such as scheduling, tokenization, serialization and KV-cache management can put pressure on the memory subsystem. That makes memory bandwidth, bandwidth per core, cache behavior and memory locality relevant to how efficiently the system advances.” “Waiting for a tool response is different. The CPU can schedule other work while one agent waits on a database, API or sandboxed application. At low concurrency, that dependency has a large effect on an individual task's completion time. At higher concurrency, the ability to run other agents makes throughput, memory bandwidth and scheduling efficiency increasingly important.” At scale, these measurements converge on the economics of the rack. Datacenters operate within fixed power, cooling, and physical constraints. The relevant question is no longer simply how much performance one processor can deliver, but how much useful agent work the infrastructure can sustain within those limits. Ultimately, what happens at the rack-level – and perhaps not too far into the future, what happens at the row-level – is what those implementing agentic AI will care about most. “That is why metrics such as tasks per rack and energy per task become important alongside CPU-level performance,” says Defilippi . “More completed tasks within the same rack-level power envelope can translate directly into better infrastructure economics and lower task cost. Thermal headroom also gives the system capacity to absorb bursts of activity without compromising performance when demand rises. We would qualify the comparison with ‘scheduled batches,’ however, because some agentic workloads can themselves be batch-oriented. The broader point is that persistent, highly utilized agent services make sustained efficiency, density, and throughput per rack increasingly important, regardless of whether an individual task is interactive or batch.” Arm has begun applying this framework to its AGI CPU. Using workloads aligned with the agentic methodology, Arm has sanctioned an independent, third party to evaluate AGI CPU performance against AMD EPYC Turin across CPU-visible parts of agentic workflows including tooling, orchestration and reasoning and planning. The testing uses measured server-level performance as the basis for rack-level modeling and the results show approximately 2X the performance per rack compared with AMD EPYC 9755, demonstrating why performance at the rack level can tell a different story from an isolated core or socket benchmark. When all is said and done, this may be the most complex capacity planning job that IT organizations have ever encountered. And they are going to need lots of help from component vendors and system makers to sort it all out because winging it and hoping for the best will be a very expensive proposition. Sponsored by Arm