I compared Qwen3.6-35B-A3B (with MoE expansion) vs Qwen3.8-27B dense at 32K and the dense model lose on GPQA-Diamond — not convinced by my own results, need help
retained text
Hi all, I'm the author of the `moe-expansion` branch in `llama.cpp` (vagrillo). I ran a comparison on **GPQA-Diamond (all 198 questions)** between: * Qwen**3.6-35B-A3B**, Q8\_0, **expanded routing** → **84.34**% (167/198) * Qwen**3.8-27B dense**, Q8\_0 → **83.84%** (166/198) * Qwen3.6-35B-A3B, Q8\_0, native routing → 81.82% (162/198) Greedy, **32K budget**, same prompt, one RTX 5000 PRO 48GB, one run per config. **Why I'm not convinced:** * 35B expanded vs 27B dense is **1 question apart** — that's parity, not a win. * Expansion vs native on the 35B is 12 wins / 7 losses (net +5), but **not significant** at n=198. * 10 of the 12 expansion wins are **Chemistry** only; Physics is saturated, Biology flat. **What I'd like help with:** 1. What other benchmarks pair well with GPQA-Diamond for this kind of routing comparison ? 2. How many runs / what setup would you consider enough for a paired comparison? 3. Is a Chemistry-only effect plausible, or a red flag? Thanks — I'm **not claimin**g this is a result (*the difference it's only 1 answer*), I'm trying to understand my own numbers **the comparison results can be found here:** [https://htmlpreview.github.io/?https://github.com/vagrillo/llama.cpp/blob/moe-expansion/benchmark/RUN1209/finalcompariso1209.html](https://htmlpreview.github.io/?https://github.com/vagrillo/llama.cpp/blob/moe-expansion/benchmark/RUN1209/finalcompariso1209.html)
document fetched 2026-09-13 00:37 · canonical https://reddit.com/r/LocalLLaMA/comments/1wej29k/i_compared_qwen3635ba3b_with_moe_expansion_vs