loading
loading
Power is a defining constraint for AI factories. As AI workloads demand a full compute platform to serve them, each component of that platform must maximize output within the factory’s limited power budget. This makes performance per watt—rather than raw, unnormalized throughput—the ultimate measure of an AI platform’s value. The NVIDIA Vera Rubin platform is designed to enable power-efficient AI at scale. At its core is NVIDIA Vera Rubin NVL72, which delivers strong performance per watt across the widest range of AI compute demands—from throughput-optimized large batches to the small batches of high-interactivity tiers, for both open and closed models. Factory and rack-level power management innovations drive Vera Rubin performance on this important metric. At the factory level, NVIDIA DSX MaxLPS software shifts power between racks as workloads demand shifts, allowing an operator to recover stranded power and provision up to 40% more GPUs within the same site-power envelope and deliver 35% higher token throughput. Inside each Vera Rubin NVL72, rack-level capacitors, along with state-of-charge Intelligent Power Smoothing software, absorb the bursty power spikes of training and inference workloads, so the factory can be planned around sustained demand instead of worst-case peaks. This means more deployable compute per megawatt. The highest interactivity tiers present unique challenges. For these, the platform adds NVIDIA Groq 3 LPX as a low-latency accelerator. This post explains innovations within individual LPX racks that make it a power-efficient contributor to the Vera Rubin platform, including its deterministic execution model. What are the power-efficiency technologies in the NVIDIA Groq 3 LPX rack? The Groq 3 LPX deterministic execution model allows the LPU compiler to create a schedule of exactly when each piece of data will move to which specific compute unit and precisely when that operation will execute, down to the clock cycle. It extends to all 256 LPU chips in the rack, enabling ultrafast interactivity at long context and power management techniques that take advantage of this determinism. After the LPU compiler has generated the workload execution schedule, it can predict the electrical current draw for each cycle in that schedule. This in turn enables two complementary technologies: - Preemptive Power (PEP): Prepares the power-delivery system for changes in demand before they arrive - Clock Period Synthesis (CPS): Shapes how abruptly demand rises and falls Together, PEP and CPS help reduce the “voltage guardband” or “electrical safety margin” that must be continually provided to any set of chips, despite not directly powering the AI workload. Decreasing this voltage guardband means a greater proportion of scarce power can be spent on the workload. What are the challenges involved in delivering power to AI workloads exactly when needed? To run AI workloads, chips execute a series of instructions ranging from simple data reshapes to complex, power-intensive matrix multiplications. Nearly all chips include hardware features that enable dynamic switching between these myriad operations while the workloads are running, as resources become available. This flexibility means that a compute-intensive operation may be scheduled to be executed at any moment. The system that delivers current to the chips must be designed to handle this. The matrix multiplies and vector multiplies that power AI workloads involve a high amount of transistor switching in a short period of time, which increases the chip’s current demand on a nanosecond scale. The chip draws that extra current from the closest available source: capacitors that sit next to the chip known as decoupling capacitors (or decaps). Discharging these decreases the chip’s voltage. The chip’s board does have a voltage regulator—a dedicated circuit whose job it is to hold the chip supply voltage at a set target—but it cannot respond instantaneously due to inductance, which resists any rapid rise in current it delivers. Nevertheless, the current the voltage regulator supplies eventually catches up with the chip’s demand, allowing the voltage to recover. These temporary voltage drops, ultimately driven by sharp changes in the current demand from the AI workload, are known as voltage droops. Their magnitude depends on how much the current changes, and on di/dt, the rate of change of the current. All else equal, larger di/dt causes a larger voltage droop; conversely, the same absolute change in current over a longer period of time will result in lower di/dt and thus less droop. These droops are usually not a problem. However, all chips have a minimum voltage (Vmin) below which the chip ceases to operate normally causing incorrect results. To reduce the likelihood of this worst-case scenario, chips are operated with a voltage guardband, providing enough supply voltage margin so the chip still receives its required minimum voltage under transient and worst-case conditions. Most of the time, the power needed to provide this guardband is effectively excess. Moreover, power is proportional to the square of voltage, so 10% more voltage continually supplied 21% more power. How does NVIDIA Groq 3 LPX cycle-exact schedule make current demand predictable? The Groq 3 LPX deterministic execution model allows the creation of a schedule for how an AI workload will run, including both operations and data movement, prior to beginning the workload. Individual LPU accelerators feature a relatively small number of distinct hardware elements, each of which enables predictably fast execution of the most common operations: - MXM for matrix multiplication - VXM for vector operations - SXM for transposing and reshaping Individual accelerators also feature hardware that keeps these compute units synchronized down to the clock cycle. For memory, each chip has hierarchy-free on-chip SRAM banks. At the LPX system level, LPUs are connected directly to each other rather than through an intermediary, making data transfer times more predictable. Determinism ties these various hardware elements together: individual computations, memory reads, and interchip communications require the same number of clock cycles from run to run. The compiler can exploit this to plan a schedule of when data will need to move between these compute elements, arriving just where it needs to, when it needs to. This schedule of compute and data movement also allows the compiler to resolve common resource conflicts, such as two hardware elements writing to the same memory bank, in advance. This cycle-exact schedule also allows the creation of the workload’s current demand over time. The compiler can estimate how much current the system will draw, at the clock cycle level. How does the execution model enable proactive voltage and clock control? The cycle-exact schedule and resulting current-demand curve enable Groq 3 LPX to reduce voltage droop and operate with a smaller voltage guardband. Because the compiler knows when current demand will rise and fall, down to the clock cycle, it can prepare the power-delivery system in advance and shape the sharpest changes in demand through the complementary PEP and CPS technologies. - PEP is the mechanism through which the chip commands the power delivery network (PDN)—the physical electronics that deliver power to the LPUs—to change the voltage it delivers. Because the compiler knows which cycle current will spike on, it can schedule that command early enough that the supplied voltage is already moving before the demand arrives. The decoupling capacitors thus have less of a gap to make up, and the chip’s voltage drops less sharply when the current ramps up. - CPS enables the compiler to schedule individual clock cycles in a workload to be shortened or lengthened. The cycle-by-cycle scheduling is only possible because of the Groq 3 LPX plesiosynchronous clock system, which keeps clocks synchronized between chips and compute elements synchronized within chips. It allows for more fine-grained control over the sharpest changes in current than PEP. In particular, the clock cycles with the very highest spikes in current can be lengthened, lowering di/dt, the rate at which current increases. Together, PEP and CPS give the compiler tools to schedule electrical signals and even clock-cycle length in ways that lead to less voltage droop. Internal testing on Groq 3 LPX systems has shown that this approach leads to >60% less voltage drop. It is estimated that these will lead to a high single-digit percentage decrease in the baseline voltage the electrical system must continually provide to AI workloads. This will contribute to even higher percentage decreases in power because power is proportional to the square of voltage—all without impacting the workload. Delivering more performance per watt—at high interactivity Groq 3 LPX deterministic execution can reduce the power required to run the same workload by a potentially low-double-digit percentage compared with a similarly specified, nondeterministic system. In a power-limited AI factory, lowering that overhead leaves more of the fixed power budget available to produce tokens. Groq 3 LPX brings these determinism-enabled power controls to the NVIDIA Vera Rubin platform in H2 2026. They complement NVIDIA DSX MaxLPS at the factory level and Intelligent Power Smoothing within the Vera Rubin NVL72 rack. Each operates at a different level of the platform, but all serve the same goal: getting more useful AI inference from every megawatt. At the platform level, pairing Groq 3 LPX with Vera Rubin NVL72 enables up to 35x higher throughput per megawatt compared to previous-generation NVIDIA GB200 NVL72 for 2T+ parameter models at long context and high interactivity. For AI factories, that means serving far more tokens within the same power budget at that demanding operating point. Acknowledgments This work was made possible through the expertise and engineering contributions of Ashraf Essea, Kibibi Moseley, Graham Steele, Suhas Somnath, Sarah McKenney, Farshad Ghodsian, and Eduardo Alvarez.
document fetched 2026-09-16 02:54 · canonical https://developer.nvidia.com/blog/how-nvidia-groq-3-lpx-deterministic-execution-drives-power-efficient-high-interactivity-inference-on-nvidia-vera-rubin