loading
loading
Off the Wire Press Releases Latest version of the benchmark introduces two new tests for emerging AI deployment patterns, including Agentic Inference SAN FRANCISCO, Sept. 16, 2026 — MLCommons announced new results for its industry-standard MLPerf Inference v6.1 benchmark suite. This release, which set a new high-water mark for the number of submitting organizations, introduces two new tests aligned with recent AI inference deployment trends. It also features the first peer-reviewed performance results for several recently released or soon-to-be-released AI platforms, demonstrating up to a 5.7X performance gain compared to just one year ago. The open-source MLPerf Inference benchmark suite measures system performance in an architecture-neutral, representative, and reproducible manner. It creates a level playing field for competition, driving innovation, performance, and energy efficiency across the entire industry. The published results provide critical technical information, allowing customers who are procuring and deploying AI systems to make informed decisions based on trusted empirical data. Evolution of the Benchmark: New Tests, New Support For Popular Optimizations MLPerf Inference v6.1 introduces two new tests to the suite that reflect the industry’s evolution towards more complex, multi-step, and agentic deployments of AI inference technology, both in the datacenter and at the edge. The End-to-End Retrieval-Augmented Generation (RAG) benchmark tests a complex pipeline of multiple distinct AI models for question answering. This workload decomposes the task into a sequence of steps: an embedding model turns a query into a vector, a retriever pulls candidate passages from a vector database, a re-ranker refines the list, and one or more LLMs reason over the data to produce an answer. The test generates performance data for two separate but related tasks: ingesting a document corpus to create a vector database, and query-answering against a pre-built vector database. More information on the End-to-End RAG benchmark is available here. The Edge Agentic Inference benchmark reflects the shift from single-shot interactions toward complex multi-turn workloads such as agentic coding. Unlike single-shot inference, agentic workloads feature growing conversational history in which each query depends on those that came before, and the model merges evidence gathering with subsequent reasoning over that evidence, iterating as necessary to arrive at a final result. This is a much more powerful usage model and unlocks tremendous new capabilities that single-shot inference lacks, while radically increasing demands on AI compute. Agentic AI deployment is increasingly popular at the edge, where serving one user at a time with fixed memory, processing, power, and context creates unique performance challenges. This test adopts the framework and methodology from the soon-to-be-released MLPerf Agentic datacenter performance benchmark and specializes it with an edge model and quantization, a single-stream coding workload, latency metrics, and a statistically robust accuracy gate. It incorporates two tasks: measuring workload accuracy under time constraints and measuring the performance of a deterministic workload in which accuracy checks are intrinsic to the workload. More information on the Edge Agentic Inference benchmark is available here. MLPerf Inference 6.1 also adds support for a performance optimization now in common use in production inference systems. Speculative Decoding, a popular optimization technique that predicts and verifies multiple tokens in a single forward pass, is now supported in the interactive scenario for two of the inference benchmarks, with added support in the GPT-OSS task. “We are working hard to ensure that the MLPerf Inference benchmark continues to reflect the scenarios that the AI community values most,” said Miro Hodak, MLPerf Inference working group co-chair. “We added the End-to-end RAG test because it’s clear that query-answering has evolved beyond simply an LLM trained on a corpus; stakeholders need to understand the real-world performance of the types of multi-step, multi-component pipelines that are being built today. Likewise, we added the Edge Agentic Inference test because complex inference systems with agentic properties are increasingly hosted on edge computing devices, creating a new set of performance challenges our customers face today. We are committed to providing timely and relevant performance data that reflects real-world production systems and the performance optimizations that are being deployed today.” Submissions and Results Demonstrate Speed of AI Innovation This round of the benchmark highlights the rapid evolution of AI technology and deployments, with submitted systems representing a range of hardware options across accelerators, boards, workstations, and servers. The results include five new processors or accelerators: AMD Ryzen AI Max+ 395, AMD Instinct MI350P, and Intel Arc Pro B70 are all available, and NVIDIA Rubin and NVIDIA Vera Rubin NVL72 is in preview. They also include the largest system ever submitted to the MLPerf Inference benchmark, with 512 accelerators and two novel heterogeneous systems. The first heterogeneous system combines high-performance networking with accelerators from two different vendors, and the second system is geographically distributed – spanning the Pacific Ocean. Performance gains continue to compound and shatter records for AI accelerator capabilities. For the Visual Language Model (VLM) test, the best per-accelerator server scenario result for this benchmark round improved by 2.99x from v6.0 just six months ago. For the Deepseek R1 test, the best per-accelerator server scenario result was 5.7x better compared to v5.1 one year ago. This performance gain translates into more capabilities for end customers, potentially serving more users or enabling more sophisticated inference use cases. “With the critical data from the MLPerf Inference benchmark, the AI community is once again proving that what can be measured can be improved,” said Frank Han, MLPerf Inference working group co-chair. “As workloads and the underlying systems increase in capability, scale, and complexity across a broader variety of form factors and physical configurations, we are clearly seeing a renewed interest in ensuring that inference can take full advantage of hardware, software, and architectural advances – both in the datacenter and at the edge. With performance data from the Inference v6.1 benchmark, customers can better understand the cost-benefit tradeoffs and make informed decisions on how to procure and deploy their AI systems.” The AI Community Continues to Embrace and Invest in MLPerf Inference The MLPerf Inference 6.1 benchmark received submissions from a record-high 30 participating organizations: AMD, ASUSTeK, Atlas Inference, Cisco, CoreWeave, Crusoe, Dell, Fujitsu Limited, GigaComputing, Google, Hewlett Packard Enterprise, Intel, Inventec Corporation, KRAI, Lambda, MangoBoost, Microsoft Azure, MiTAC, Nebius, NVIDIA, Oracle, Orrick, Quanta Cloud Technology, RedHat, ScitiX, Supermicro, Telecommunications Technology Association, VibeHPC, Wiwynn and individual contributor Naeem Khoshnevis. “I would like to welcome our six first-time submitters: Atlas Inference, Crusoe, Orrick Industries LLC, ScitiX, VibeHPC, and individual contributor Naeem Khoshnevis, ” said David Kanter, Head of MLPerf. “The groundswell of participation from the AI community tells us that our work is important and is making a difference to stakeholders.” In addition, over 50% of submitters used MLPerf’s new API-centric harness, the foundation of the next-generation MLPerf Endpoints benchmark suite, in their submissions. The new harness uses a true client/server architecture over industry-standard APIs to communicate inference queries and results to the system under test, more accurately reflecting real-world datacenter deployment. “We’re excited to see submitters embrace our customer-centric tooling and workloads and partner with us to improve the state of industry-wide benchmarking for AI,” said Kanter. “Moving forward, MLPerf Endpoints will replace Inference in our family of benchmarks for the datacenter, and the quick uptake of our API-centric harness will contribute to making that transition seamless.” View the Results To view the results for MLPerf Inference v6.1, please visit the Datacenter and Edge benchmark results pages and read details on individual submissions in our supplemental. For an interactive view of datacenter results, access the MLPerf benchmark results dashboard: https://mlcommons.org/visualizer. Additionally, submitters have provided supplemental statements to help contextualize and understand their submissions and results. About MLCommons MLCommons is the world’s leader in AI benchmarking. An open engineering consortium supported by over 130 members and affiliates, MLCommons has a proven record of bringing together academia, industry, and civil society to measure and improve AI. The foundation for MLCommons began with the MLPerf benchmarks in 2018, which rapidly grew into a set of industry metrics for measuring machine learning performance and promoting transparency in machine learning techniques. Since then, MLCommons has continued to use collective engineering to build the benchmarks and metrics required to evaluate AI systems’ accuracy, safety, speed, and efficiency, so that customers can understand the tradeoffs and make informed decisions. Source: MLCommons
document fetched 2026-09-17 02:48 · canonical https://hpcwire.com/off-the-wire/mlcommons-sets-participation-record-with-new-mlperf-inference-v6-1-benchmark-results