← sources

Research paper: Parallelism Strategies and Concurrency Effects for Mixture-of-Experts Inference on GPU Systems

source: semantic-scholarpublished 2026-05-01 00:00ingested 2026-07-02 08:48

retained text

Query: mixture of experts inference serving
Authors: Ananya Hegde, Akshata Kumble, Ravi Gupta
Citations: 0
Mixture-of-Experts (MoE) architectures reduce inference cost by activating only a sparse subset of parameters per token. However, when these models exceed single-GPU memory, practitioners must distribute them across devices using tensor parallelism (TP), expert parallelism (EP), or a hybrid of both, and there is little empirical guidance on which strategy to choose. We present a controlled comparison of these strategies for MoE inference serving on NVIDIA A100 GPUs using vLLM. Across multiple models and workload settings, we find that EP achieves 85–102% of TP throughput depending on expert count, matching or exceeding TP for models with 60–64 experts, while naive TP+EP hybrid placement underperforms both alternatives. We also show that a diffusion-based MoE language model is substantially slower than its dense counterpart, highlighting the need for multi-GPU expert distribution in this emerging class. Overall, our results suggest that request concurrency has a larger impact on throughput than parallelism choice in many cases, and that EP is a strong default for large-expert MoE models.

document fetched 2026-08-10 07:17 · canonical https://semanticscholar.org/paper/194facd3a3e40fdf53e4e5e52205e83c4162224d