← sources

Research paper: Beyond Throughput: Performance and Energy Insights of LLM Inference Across AI Accelerators

source: semantic-scholarpublished 2026-05-25 00:00ingested 2026-07-25 07:53

retained text

Query: large language model inference gpu datacenter
Authors: Giacomo Brunetta, V. Sastry, Xingfu Wu, Valerie Taylor, Michael E. Papka
Citations: 0
Serving large language models (LLMs) is highly resource-intensive and requires specialized hardware acceleration. While GPUs remain the dominant solution, dataflow accelerators are emerging as a compelling alternative. In this study, we present an extensive empirical performance analysis of six datacenter-class GPUs from Nvidia, AMD, and Intel, alongside two dataflow AI accelerators from Cerebras and SambaNova, using fourteen different open-source LLMs. Our evaluation focuses on investigating the main factors that impact LLM inference, including model size, batch size, quantization, and multi-GPU scaling under various parallelism strategies. Importantly, we analyze both performance and energy efficiency, providing an energy-aware comparison across accelerator types. Our experimental results show that dataflow AI accelerators achieve an order-of-magnitude improvement in throughput and latency for small (≤ 8) batch sizes compared to GPUs. On the other hand, GPUs offer larger HBM memories and benefit from a simpler programming model, allowing for more flexibility in terms of batch size, which is particularly beneficial in offline inference. These insights provide practical guidance for LLM inference in diverse deployment scenarios.

document fetched 2026-08-06 08:19 · canonical https://semanticscholar.org/paper/27677d62299728ddbb17a7d972dff7a4f3a43dea