# Qwen3-0.6B model converted into MoE — retained record

> Retained reddit record from reddit:LocalLLaMA.

Canonical: https://highsignal.app/data/records/59a66469f3b04a98

[← sources](https://highsignal.app/data)


source: [reddit](https://highsignal.app/data/reddit) (reddit:LocalLLaMA)  published 2026-10-23 08:18 ingested 2026-10-24 02:40

[original source ↗](https://www.reddit.com/r/LocalLLaMA/comments/1xd87wu/qwen306b_model_converted_into_moe/)

## retained text



```
This is another update about my MoE-fication project, the first post can be found [here](https://www.reddit.com/r/LocalLLaMA/s/4EyuGKOSpx).

# Foreword

Since my last model conversion, I’ve been experimenting a lot with new methods, though sadly, most attempts haven't helped much. Compatibility has been the main limiting factor. I’ve made GGUF compatibility a priority to ensure as many people as possible can use my models, but this means sacrificing some potential improvements.

Still, my perfectionism has finally let me publish another model: Qwen3-0.6B, my first chat-style conversion. This brought two new challenges: preserving its chat capabilities during conversion and adapting my methods from Qwen 2.5’s `qwen2moe` to Qwen3’s `qwen3moe`.

The latter proved particularly difficult. Unlike the previous architecture, Qwen3 has no shared expert, which helped preserve quality in my earlier conversions by keeping important neurons always active. Without it, quality dropped considerably.

To mitigate this, I experimented with a pseudo-shared expert, basically an expert that the router always picks. It doesn’t work quite as well as a true shared expert due to differences in weighting, but it does recover some of the lost quality.

I’m also considering releasing a separate set of conversions that prioritise quality over GGUF compatibility, allowing techniques like dense-layer substitution.

Finally, I’d like your thoughts on another experiment: duplicating neurons to create new expert pairings. Early results showed significant improvements, but at the cost of doubling the total parameter count which I think kind of defeats the purpose of the conversion. What do you guys think?

# New Model

So my latest conversion was of the [Qwen/Qwen3-0.6B](https://huggingface.co/Qwen/Qwen3-0.6B). The conversion can be found here [bayliner1980/Qwen3-0.6B-MoE-A0.5B](https://huggingface.co/bayliner1980/Qwen3-0.6B-MoE-A0.5B) with GGUF version here [bayliner1980/Qwen3-0.6B-MoE-A0.5B-GGUF](https://huggingface.co/bayliner1980/Qwen3-0.6B-MoE-A0.5B-GGUF).

# Evaluation

||Wiki ppl|C4 ppl|Tasks|
|:-|:-|:-|:-|
|Qwen3-0.6B dense|23.88|31.63|0.455|
|Qwen3-0.6B MoE|26.98|36.32|0.425|

# Benchmark

||GPU generation|CPU generation|
|:-|:-|:-|
|Dense|377 t/s|115 t/s|
|MoE|352 t/s|124 t/s|

At lower parameters like 0.6B there is no improvement to speed with conversion. Only on larger models like my 1.5B conversion is there actually any benefit.

# Limitations

* It gets specific facts confidently wrong. Asked who painted the Mona Lisa and when, it says Leonardo da Vinci in 1519 (that's the year he died).
* Code is broken
* Use non-thinking mode; thinking was never trained after conversion.
* Worse than the original on every quality metric. If you just want a small model, use Qwen3-0.6B.

# Support

If you find my stuff interesting consider supporting me on [Ko-fi](https://ko-fi.com/bayliner1980).
```



document fetched 2026-10-24 00:42 · canonical https://reddit.com/r/LocalLLaMA/comments/1xd87wu/qwen306b_model_converted_into_moe
