MLPerf Inference v6.1: 5.7x Per-Accelerator Gains, a 512-GPU Run, and Vera Rubin’s First Peer-Reviewed Numbers

retained text

MLCommons has published MLPerf Inference v6.1, and the round sets a participation record with 30 submitting organizations and 486 datacenter and edge results. Two new tests join the suite: an End-to-End RAG pipeline for the datacenter and an Edge Agentic Inference benchmark for single-user devices, and the results carry the first peer-reviewed numbers for NVIDIA’s Vera Rubin NVL72, AMD’s Instinct MI350P, Intel’s Arc Pro B70, and AMD’s Ryzen AI Max+ 395. On the pace of improvement, MLCommons says the best per-accelerator DeepSeek-R1 result in the server scenario is 5.7x better than in v5.1 a year ago, and the best VLM result improved 2.99x in the six months since v6.0.
Two New Tests: End-to-End RAG and Edge Agentic Inference
The End-to-End RAG benchmark measures a complete question-answering pipeline, several models and a vector database working together. The reference implementation runs four models together: gpt-oss-120B handles query decomposition, sufficiency checking, and answer generation; gpt-oss-20B grades retrieved documents; e5-base-v2 produces embeddings; and ColBERTv2 reranks passages. The corpus comprises 107,484 passages, chunked from 2,515 HTML files; the questions are 824 multi-hop tasks from Google’s FRAMES dataset; and each task can loop through up to 5 retrieval rounds before the pipeline decides it has enough evidence. Two metrics come out: documents per second for building the FAISS HNSW vector database, and tasks per second for answering questions against it. A Llama 3.1-8B judge scores the final answers against a 97% accuracy target, and the judging isn’t timed.
The Edge Agentic Inference benchmark targets the coding-assistant pattern that has moved onto workstations and desktop AI boxes. The model is Qwen3.6-27B with thinking off, run as a Q4_K_M GGUF under llama.cpp in the reference, with a 32K context window served per turn. The performance workload is a recorded replay of 20 agentic coding trajectories drawn from SWE-bench Verified, totaling 1,007 turns, driven in a single stream with one request in flight, the way a developer on a laptop runs an agent. The reported metric is mean latency per turn, with time-to-first-token and time-per-output-token distributions alongside, and accuracy is gated separately by BFCL v4 at 97% of the reference score. MLCommons adapted the framework from its upcoming MLPerf Agentic datacenter benchmark.
“We added the End-to-end RAG test because it’s clear that query-answering has evolved beyond simply an LLM trained on a corpus; stakeholders need to understand the real-world performance of the types of multi-step, multi-component pipelines that are being built today,” said Miro Hodak, MLPerf Inference working group co-chair. “Likewise, we added the Edge Agentic Inference test because complex inference systems with agentic properties are increasingly hosted on edge computing devices, creating a new set of performance challenges our customers face today.”
The round also extends speculative decoding support, previously limited to DeepSeek-R1, to the GPT-OSS benchmark in the interactive scenario, and defines a new interactive scenario for the VLM test with responses targeted at about 1.5 seconds.
New Silicon From the Desktop to the Rack
NVIDIA’s Vera Rubin NVL72 makes its MLPerf debut in the preview category, submitted by NVIDIA and by Nebius on its VR200 NVL72. NVIDIA reports up to 2.5x higher token throughput than the GB300 NVL72 on DeepSeek-R1 across offline, server, and interactive scenarios using TensorRT-LLM, and up to 3.7x on the Qwen3 vision-language model using vLLM with NVIDIA Dynamo. Those are NVIDIA’s comparisons against its own prior generation, and the preview designation means the platform is expected to be commercially available by the next round.
AMD expanded its MLPerf Inference 6.1 submission to six model families across language, reasoning, text-to-video, and recommendation tasks, deploying the Instinct MI355X, MI350X, and the new MI350P PCIe card. The 512-GPU Crusoe cluster built on the MI355X is covered below, along with AMD’s own breakdown of the round.
Intel’s Arc Pro B70 shows up in a four-GPU node with 128GB of combined VRAM that Intel used for Llama 3.1-8B, Llama 2-70B, gpt-oss-120B, Whisper, and the new E2E-RAG test; Intel reports gpt-oss-120B improved 36% in server and 27% in offline over v6.0 on the same hardware. On the CPU side, Intel says Xeon 6980P Llama 3.1-8B server throughput rose 2.4x from v6.0 on identical silicon, a software-only gain.
The Ryzen AI Max+ 395 appears through Atlas Inference, a first-time submitter that ran the new Edge Agentic workload on both an NVIDIA DGX Spark and an AMD Strix Halo desktop with the same engine and quantization recipe. Atlas reports 20.1 tokens per second on the DGX Spark, completing all 1,007 turns in under 64 minutes, and 19.63 tokens per second on Strix Halo. That’s a narrower gap than we measured between the two platforms with off-the-shelf runtimes in our Ryzen AI Halo and DGX Spark reviews, and it’s the kind of result the new benchmark is designed to surface.
Bigger, More Diverse, and More Distributed
Multi-node submissions hit a record this round, up from zero in v4.0, and three stand out. Crusoe, another first-time submitter, ran the largest system in MLPerf Inference history with AMD: 512 Instinct MI355X GPUs across 64 nodes on a standard RoCE Ethernet fabric, submitted for gpt-oss-120b and DeepSeek-R1. AMD reports 5.75 million tokens per second in the offline scenario and 5.39 million in server on gpt-oss-120b from that cluster, and 2.90 million offline and 2.41 million server on DeepSeek-R1, which AMD calls the highest aggregate token throughput in MLPerf history, with throughput scaling near-linearly from 1 to 64 nodes. The gpt-oss-120b run served the model as 512 independent single-GPU replicas in native MXFP4; DeepSeek-R1 used SGLang with one eight-GPU replica per node. AMD separately cites a 72-GPU GPT-OSS-120B submission at 95% scaling efficiency and 1 million tokens per second, the same headline it hit on MI355X in v6.0.
Cisco submitted the benchmark’s first cross-vendor heterogeneous system, pooling eight NVIDIA H200 and eight AMD Instinct MI350X GPUs into a single inference pool over a Cisco Silicon One G200 fabric, the same mixed-accelerator approach it’s selling through its Secure AI Factory. The geographically distributed entry came from MangoBoost with Dell: four sites on two continents, hosted by MangoBoost, Dell, TensorWave, and Microsoft Azure, spanning the Pacific and running as one endpoint at what MangoBoost reports as 97% scaling efficiency. MangoBoost also claims the first prefill/decode-disaggregated results on AMD Instinct GPUs.
Elsewhere in the results, CoreWeave reports 1.16 million tokens per second aggregate on GB300 NVL72 with per-GPU offline throughput up 17% since v6.0, HPE cites 8,500 tokens per second per GPU on DeepSeek-R1 across two Compute XD690 servers with Blackwell Ultra, and Google focused its submission on DeepSeek-R1, citing the industry’s shift to large mixture-of-experts models. gpt-oss-120b drew 112 submissions, the most of any MLPerf workload.
AMD’s Two CDNA 4 Form Factors: OAM and Dual-Slot PCIe
AMD’s CDNA 4 architecture is available in two physical form factors to meet specific data center power and cooling constraints. The flagship Instinct MI355X targets high-density compute nodes using an OAM form factor on an OCP Universal Baseboard (UBB 2.0) platform, providing 256 compute units, 288 GB of HBM3E memory, and 8 TB/s of aggregate memory bandwidth. It delivers up to 10.1 PFLOPS of peak theoretical MXFP4 and MXFP6 matrix compute. The newly introduced Instinct MI350P adapts that same CDNA 4 silicon into a dual-slot PCIe 5.0 add-in card for standard enterprise chassis, housing 128 compute units, 144 GB of HBM3E memory, 4 TB/s of bandwidth, and delivering up to 4.6 PFLOPS of peak theoretical MXFP4 and MXFP6 matrix performance.
Software-Driven Generational Uplift on Identical Silicon
Software optimizations in AMD ROCm v7 yielded measurable throughput gains on identical MI355X hardware during a single MLPerf cycle. Testing on an eight-GPU MI355X node showed GPT-OSS-120B throughput rose by 28% in the Offline scenario and 38% in the Server scenario, while Wan-2.2 SingleStream performance improved by 70%. At cluster scale, 72 MI355X accelerators in MLPerf 6.1 achieved higher aggregate GPT-OSS-120B throughput than a 94-GPU configuration reported in round 6.0. In competitive comparisons published by AMD, the eight-GPU MI355X led submitted results against the NVIDIA B200 and B300 on GPT-OSS-120B, while the 72-GPU cluster led NVIDIA’s GB200 submission.
In its first MLPerf round, the dual-slot MI350P submitted across five Closed workloads, posting leading results against selected submissions of the NVIDIA RTX PRO 6000 Server Edition and H200 NVL. For deployments sensitive to power and cooling budgets, an eight-GPU MI350X system maintained approximately 80% of the MI355X platform’s benchmark performance across GPT-OSS, Llama, Wan, and DLRM workloads, while the MI355X carries a 40% higher rated TDP. A commissioned study by Signal65 reported that these throughput numbers translated to lower operational cost per document and higher token output per dollar within fixed latency limits.
AMD Partner Results and the Korea-to-US Cluster
AMD says comparable MI355X submissions from seven partners, Dell Technologies, Oracle, Hewlett Packard Enterprise, Supermicro, MangoBoost, Crusoe, and MiTAC, averaged within 4% of its reference system results, with some runs matching or slightly exceeding them.
The MangoBoost and Dell entry noted above is the benchmark’s first heterogeneous 32-GPU serving configuration, bridging 16 previous-generation Instinct MI300X GPUs in Korea with 16 Instinct MI355X GPUs in the United States into a unified GPT-OSS-120B endpoint. The split-cluster configuration delivered 285,454 Offline tokens per second and 253,501 Server tokens per second. The submissions ran on ROCm 7; AMD points to the ROCm 10 release, with vLLM, SGLang, and ROCm.AI profiling tools, as the path forward ahead of the HBM4-based MI400 Series and the MI500 generation that follows.
The Harness That Replaces MLPerf Inference in the Datacenter
Sixteen of the 30 submitters used MLPerf’s API-centric harness this round, up from a single open-division submitter in v6.0. The harness runs a true client/server setup over standard APIs against a hosted endpoint, which is how datacenter inference is deployed, and it already carries the new Edge Agentic test, VLM-Interactive, gpt-oss, DeepSeek-R1, Llama 3.1-8B, and text-to-video. It’s the foundation of MLPerf Endpoints, which opens on-demand rolling submissions in October 2026 and replaces MLPerf Inference as the datacenter benchmark in 2027, with normalized results and expanded agentic workloads planned for Endpoints v1.0.
“Moving forward, MLPerf Endpoints will replace Inference in our family of benchmarks for the datacenter, and the quick uptake of our API-centric harness will contribute to making that transition seamless,” said David Kanter, head of MLPerf. The six first-time submitters this round are Atlas Inference, Crusoe, Orrick Industries, ScitiX, VibeHPC, and individual contributor Naeem Khoshnevis of Harvard’s Kempner Institute, who submitted a single-H200 Llama 3.1-8B result.
The inference round follows the MLPerf Storage v3.0 results published two weeks ago, and full datacenter and edge tables, along with submitter supplementals, are available on the MLCommons results pages.

document fetched 2026-09-17 02:48 · canonical https://storagereview.com/news/mlperf-inference-v6-1-5-7x-per-accelerator-gains-a-512-gpu-run-and-vera-rubins-first-peer-reviewed-numbers