Scaling NVIDIA TensorRT Performance Optimization Beyond a Single Build Route with the Global Performance Tuner, Part 1

extract

Deploying a trained neural network involves more than translating each operator into a GPU kernel. It is a compilation problem with many legitimate answers. For example, when NVIDIA TensorRT builds an engine, it makes interdependent decisions about graph-IR transformations, fusion, implementation selection, memory placement, and quantization. This process is called scheduling. Different scheduling choices can preserve model semantics yet produce very different inference performance, making scheduling central to deep learning compiler optimization. No single scheduling strategy is best for every model and deployment environment. A transformation that accelerates one network may slow another. A kernel that is faster in isolation may interact poorly with fusion, tensor layouts, or memory traffic elsewhere in the graph. Aggressive scheduling may improve performance while violating an application’s accuracy requirement. These coupled, model-dependent effects often appear only after the complete engine is built and measured, making the right strategy increasingly difficult to predict. This post is Part 1 of a two-part series that is intended to develop a clearer way of reasoning about performance optimization. It explores questions such as: What makes one compiled engine better than another? Why can individually promising decisions fail end to end? How should accuracy constrain performance optimization? It introduces the Global Performance Tuner, a feature for automatic performance tuning released in TensorRT 11.1, and explores these questions through an intuitive example and mathematical framework. What challenge does the Global Performance Tuner feature solve? The following analogy illustrates the motivation for building the Global Performance Tuner feature in TensorRT. Imagine a chef with one excellent dish that has pleased three regular customers for years. One likes spice, another sweet-and-sour flavors, and the third extra salt. The chef combines chili, syrup, lemon, and salt in one standard recipe that satisfies all three. One day, a new fourth customer arrives who dislikes spice. Keeping the chili disappoints the newcomer; removing it risks losing the first customer. The root cause of the chef’s dilemma is the method of adding the seasonings does not scale out: the number of interacting exceptions grows with the customer base, and a change for one customer can degrade the meal for another. A scalable approach keeps the base dish but makes the seasonings configurable. The chef records an appropriate combination for each regular customer. For a new customer with unknown preferences, the chef offers free samples with different combinations, rejects unsuitable variations, and keeps the one the customer likes best. Mapping this analogy to deep learning compilation, TensorRT is the chef, its shared pipeline the base recipe, tunable compiler configurations the seasonings, and the compiled engine the finished dish. Each ONNX model is a customer. Performance tuning offers samples, and end-to-end performance reflects customer satisfaction. Historically, TensorRT performance optimization has started with internal KPI models. Engineers study these workloads, identify optimizable patterns, and encode their findings as heuristics in the TensorRT default scheduling strategy. Each heuristic captures expert knowledge and allows users to benefit automatically from optimizations discovered on representative models. As models and deployment environments diversify, this approach leads to a scaling problem similar to the one described in the analogy. A heuristic inspired by one internal KPI model joins the shared strategy and may help familiar patterns while causing a regression for an unseen external model. Adding another check or exception may protect that model, but repeating the process makes the scheduling logic increasingly difficult to extend. The problem appears in three related forms: - Heuristic sprawl: Each new model-specific heuristic introduces another rule whose interactions with existing rules must be understood and maintained. As these rules accumulate, the code becomes unmaintainable, and improvements for one family of models can unintentionally affect another. - Performance-accuracy entanglement: Reduced-precision kernels and other aggressive optimizations can improve performance while introducing numerical differences that exceed an application’s acceptable tolerance. Higher-precision choices may reduce that risk but sacrifice performance. Performance and numerical acceptability therefore cannot always be optimized independently. - Performance customization: Scheduling choices that work well for one model or GPU may not work well for another. Encoding every successful choice into one universal scheduling strategy makes it difficult to specialize TensorRT efficiently for a continually expanding range of external models, hardware platforms, and application requirements. Tuning provides a scalable complement to heuristic optimization. Mathematical theory for deep learning compiler performance This section develops an abstract mathematical model of deep learning compiler performance. The purpose is to separate two fundamental questions: - What executables a compiler can construct for a model - How the compiler selects one from those possibilities for optimal performance Concept: Model, model space Let \(m\) represent a model and \(M\) represent the collection of all models under consideration, called the model space: \(m \in M\) Here, a model is an abstract description of the computation to be compiled. Its concrete representation may be an ONNX graph, a network created through the TensorRT API, a PyTorch model, or another frontend representation that fully specifies the model computation. Concept: Solution, solution space Given a model \(m\) as input, a deep learning compiler constructs an executable output for the target hardware, defined as a solution. The same model can generally be compiled into different solutions under different scheduling policies. Let \(s\) represent one such solution, and let \(S\) represent the collection of possible solutions, called the solution space. A solution is an abstraction of executable computation on the target hardware. Concretely, a solution may be a TensorRT engine or a cubin of serialized sequence of GPU kernels. More abstractly, it may also be a compiler’s intermediate representation (IR), provided that the representation uniquely determines the final executable under the remaining compilation process. Each solution also defines a function from the model’s input domain to its output codomain: \(s : X \rightarrow Y\) \(s(x) = y\) Here, \((x,y)\) is an input-output pair for the sequence of GPU kernels that implements the neural network. At this level of abstraction, the model describes the intended computation, while a solution describes one particular way of realizing that computation on the target hardware. Principle of determinism Statement: When both a deterministic program and its complete execution environment remain fixed, the program has a fixed underlying run time. This principle serves as a modeling assumption for performance analysis. Under a rigorous interpretation, the environment includes the input data, target device, software stack, GPU frequency, temperature, power state, system load, and other execution conditions that may influence runtime. In practice, not every environmental variable can be controlled perfectly, so repeated runtime measurements exhibit noise. If this disturbance is modeled as independent Gaussian noise, the average of repeated measurements is the maximum-likelihood estimation of the underlying mean run time. Once the execution environment has been fixed at the level required by the abstraction, each solution can be associated with one characteristic runtime. This association enables runtime to be represented as a mathematical function. Concept: Run time function The run time function \(T\) maps each solution to its runtime on a specified target device under a fixed execution environment: \(T(s_0) = t_0\) The environment is treated as fixed and is therefore omitted from the notation. Concept: Builder algorithm A builder algorithm (also called a performance scheduler) is a mapping from the model space to the power set of solution space. When given a specific model as input, the builder algorithm will output a solution space with a given accuracy requirement. In the accuracy condition shown above, the input–output pairs \((x, y)\) are treated as reference values, and \(ε\) represents the accuracy tolerance threshold. The reference data, comparison metric, and tolerance are treated as fixed context and are therefore suppressed from the arguments of \(B\). Ideally, the builder algorithm should generate a sufficiently rich set of solutions to contain \(s^*\), the optimal solution: the fastest feasible solution in the modeled solution space. Concept: Searching algorithm A searching algorithm (also called a solver of solution space) is a mapping from the power set of solution space to the solution space. When given a set of candidate solutions as input, the searching algorithm will pick one solution from it as the output. \(A(S) = argmin_{s_i \in \underline{S}} T(s_i), \underline{S} \subseteq S\) Note that in the real world, a searching algorithm may pick a suboptimal solution from the solution space. In TensorRT, this suboptimality can arise from at least two sources: - Heuristics used during searching: A heuristic may remove candidates before they can be evaluated as part of a complete engine. For example, the builder may filter certain convolution tactics according to a channel-dimension threshold in a certain layer of the input model. If a filtered tactic would have participated in the fastest end-to-end solution, the searching algorithm can no longer discover that optimal solution. - A mismatch between local profiling and end-to-end execution: During engine building, kernel candidates may be profiled individually during tactic selection, however, performance is measured across the complete engine when inference. A kernel running in isolation and the same kernel running as part of a larger graph do not share an identical execution environment (for example, the adaptive grid sizing technique used in an NVIDIA Blackwell GPU). Consequently, the sum of individually measured kernel runtimes does not necessarily equal the end-to-end runtime. In the ideal case, the builder algorithm generates a candidate set containing \(s^*\), and the searching algorithm reliably selects it: A deep learning compiler can therefore be represented by an algorithm tuple \((A, B)\), where \(B\) generates candidate solutions and \(A\) selects among them. In an actual implementation, however, candidate generation, pruning, profiling, and selection may be tightly coupled. The boundary between \(A\) and \(B\) may therefore not be explicit in the compiler architecture. Let \(B^*\) denote the optimal builder algorithm among the builder algorithm family, and \(A^*\) denote the optimal searching algorithm in the searching algorithm family. The ideal compiler would implement the tuple \((A^*,B^*)\). For any input model, this tuple would generate a candidate set containing the global optimal solution and then reliably discover it: At the level of the mathematical abstraction, one possible implementation of \(A^*\) is brute-force algorithm—evaluate every solution in the candidate set and select the one with the lowest runtime: \(A^*(S) = argmin_{s_i \in S} T(s_i)\) Though applying the brute-force searching algorithm will not always be acceptable when the build time budget is limited, there could be trade-offs between build time and inference performance. \(A^*\) is used to represent the searching algorithms that will definitely return the optimal solution given any input model. Constructing \(B^*\) is the more fundamental challenge. Even an ideal searching algorithm cannot select a solution that the builder never makes reachable. Rather than expecting to construct \(B^*\) in a single step, it’s possible to progressively approximate it by expanding and improving the build routes available to the compiler. Global Performance Tuner modeling This section models compilation as a sequence of compiler passes, then shows how customizable build routes provide a foundation for accuracy-aware performance tuning. Deep learning compiler modeling Like a traditional compiler, a deep learning compiler can be viewed as a sequence of passes, each of which transforms one IR into another. As previously mentioned, IR has a one-to-one mapping to the final executable binary; thus IR could be thought of as a solution. For this abstraction, let \(f\) represent an IR self-map from the solution space to the solution space: \(f : S \rightarrow S\) \(f(s) = s’\) The builder algorithm of a deep learning compiler can be modeled as a series of passes to convert one solution to another solution: The builder algorithm \(B\) is a function composite that looks like a route that the model walks through and gets transformed into a solution space, so \(B\) can also be called a build route. Note the following: - Strictly speaking, at least one compiler pass (such as the final code-generation transition) maps from model space \(M\) into executable solution space \(S\). However, compiler passes are not always perfectly decoupled, and an IR can be treated as uniquely identifying the engine produced by the remaining fixed pipeline. Modeling the passes as transformations over a generalized solution-state space therefore provides a useful simplification. - Some compiler passes are functionally indispensable while not participating in the performance choices under study. Such performance-irrelevant passes can be treated as identity mapping or omitted from the tunable portion of the route because they do not distinguish one route from another. - At first glance, the composition above may appear to imply that one build route generates only one solution (that is, the built engine). In practice, the TensorRT compilation pipeline may consider and filter many potential tactics before producing the final engine. Much of this candidate-generation process remains implicit in the composition because algorithms \(A\) and \(B\) are tightly coupled. The formula describes the selected solution, not every candidate that was considered or discarded along the way. Customizable build route Experience from performance analysis and optimization shows that effective scheduling is model-dependent. Different models can benefit from different choices in individual compiler components, even when they run on the same target GPU. Consequently, the optimal solutions for all models do not necessarily originate from one universal build route. A build route should therefore be customizable, allowing scheduling decisions that adapt to the model and its deployment environment. Such a tunable builder algorithm can be called a customizable build route. For example, consider two models, \(m\) and \(m’\), and a family of available build routes: \(\{B_1, B_2, \ldots, B_n\}\) The two models need not share the same optimal route. Model \(m\) may achieve its best result through \(B_3\), while model \(m’\) may achieve its best result through \(B_1\): \(A^*(B(m)) = \arg\min_{B_i(m)} \{T(B_1(m)), T(B_2(m)), \ldots, T(B_n(m))\} = B_3(m)\) \(A^*(B(m’)) = \arg\min_{B_i(m’)} \{T(B_1(m’)), T(B_2(m’)), \ldots, T(B_n(m’))\} = B_1(m’)\) The Global Performance Tuner begins with the idea of customizable build routes. For each candidate solution, it validates the outputs against the specified accuracy criterion and measures end-to-end inference performance. Among all solutions that meet accuracy requirements, the searching algorithm selects the minimum-runtime feasible solution from the expanded solution set. The following three principles define the framework: - Accuracy-aware tuning: Use end-to-end accuracy as a constraint that guides performance tuning. - Customizable build route: Make build routes customizable so that scheduling can adapt to each model and deployment environment. - Large solution space: Construct a rich feasible solution set, then select the fastest candidate whose numerical difference remains within the specified tolerance. The mathematical representation of the Global Performance Tuner can be written as: Learn more To learn more, check out the Global Performance Tuner documentation and read Part 2 of this series, which describes the algorithm and architecture design of the Global Performance Tuner and best practices for using it.