Fastest inference engine for MoE models on a single RTX 3090?
retained text
I've been out of the local LLM scene for a while and I'm trying to catch up with the latest developments. What's currently the fastest inference engine for running MoE models like Gemma 4 26B-A4B or Qwen3.6-35B-A3B on a single RTX 3090 (24GB)? I'm mainly interested in maximum generation speed (tokens/sec) at Q4 quantization on Windows. Any real-world benchmarks or recommendations would be appreciated!
document fetched 2026-11-20 00:42 · canonical https://reddit.com/r/LocalLLaMA/comments/1xzyr3y/fastest_inference_engine_for_moe_models_on_a