← sources

Fastest inference engine for MoE models on a single RTX 3090?

source: reddit (reddit:LocalLLaMA)published 2026-11-19 03:02ingested 2026-11-20 02:39

retained text

I've been out of the local LLM scene for a while and I'm trying to catch up with the latest developments.

What's currently the fastest inference engine for running MoE models like Gemma 4 26B-A4B or Qwen3.6-35B-A3B on a single RTX 3090 (24GB)?

I'm mainly interested in maximum generation speed (tokens/sec) at Q4 quantization on Windows.

Any real-world benchmarks or recommendations would be appreciated!

document fetched 2026-11-20 00:42 · canonical https://reddit.com/r/LocalLLaMA/comments/1xzyr3y/fastest_inference_engine_for_moe_models_on_a