# Update: Converting dense models into Mixture-of-Experts — retained record

> Retained reddit record from reddit:LocalLLaMA.

Canonical: https://highsignal.app/data/records/7dbe0d05210330bf

[← sources](https://highsignal.app/data)


source: [reddit](https://highsignal.app/data/reddit) (reddit:LocalLLaMA)  published 2026-10-13 11:50 ingested 2026-10-14 02:43

[original source ↗](https://www.reddit.com/r/LocalLLaMA/comments/1x4uint/update_converting_dense_models_into/)

## retained text



```
This is an update to my previous post which can be found here: [https://www.reddit.com/r/LocalLLaMA/s/n7sgRZZc4z](https://www.reddit.com/r/LocalLLaMA/s/n7sgRZZc4z)

Since my last post I have continued working on the conversion process and also converting new models. My latest model is the [Qwen2.5-1.5B-MoE-A0.9B](https://huggingface.co/bayliner1980/Qwen2.5-1.5B-MoE-A0.9B). I have also released GGUF versions for both the 0.5B and the 1.5B models which can be found here [bayliner1980/Qwen2.5-0.5B-MoE-A0.3B-GGUF](https://huggingface.co/bayliner1980/Qwen2.5-0.5B-MoE-A0.3B-GGUF) and here [bayliner1980/Qwen2.5-1.5B-MoE-A0.9B-GGUF](https://huggingface.co/bayliner1980/Qwen2.5-1.5B-MoE-A0.9B-GGUF).

# Evaluation

|\-|Dense|MoE|
|:-|:-|:-|
|WikiText-2 test perplexity (1024-token windows)|10.41|14.63 (+41%)|
|C4 validation perplexity (64 × 1024 tokens)|15.76|22.88 (+45%)|
|Zero-shot tasks, mean acc\_norm|0.565|0.462|

|Task|Dense|MoE|
|:-|:-|:-|
|HellaSwag|0.592|0.510|
|ARC-Easy|0.723|0.572|
|ARC-Challenge|0.452|0.311|
|WinoGrande|0.646|0.588|
|OpenBookQA|0.410|0.328|

Perplexity is about 40% higher, and the benchmarks drop the most on ARC.

# llama-bench, Q4_K_M for both, RTX 4060 8 GB / Ryzen 5 7500F

|\-|Dense|MoE|Change|
|:-|:-|:-|:-|
|GPU, text generation tokens/s|191|253|\+32%|
|CPU, text generation tokens/s|54.9|77.5|\+41%|

This is the first conversion that's clearly faster than the dense model on GPU too. The 0.5B only gained 6% there.

# Honest caveats

* At this size, quality is roughly in the range of Qwen2.5-0.5B, which is even faster. So this release is more a proof of concept. The approach should pay off more on larger models, where the MLP is a bigger share of the compute.
* The GGUF is larger than the dense one (1.1 GB vs 0.9 GB at Q4\_K\_M), because llama.cpp's MoE architecture needs an untied output head.

# Support

If you find my stuff interesting consider supporting me on [Ko-fi](https://ko-fi.com/bayliner1980).
```



document fetched 2026-10-14 00:42 · canonical https://reddit.com/r/LocalLLaMA/comments/1x4uint/update_converting_dense_models_into
