← sources

Qwengram-0.8B: I transferred Qwen3.8 Flash-Next’s n-gram memory into Qwen3.5-0.8B — 5.05% lower validation perplexity

source: reddit (reddit:LocalLLaMA)published 2026-09-25 12:46ingested 2026-09-26 02:38

retained text

I’ve been experimenting with whether Qwen3.8-Flash-Next’s pretrained PLE n-gram memory can improve a much smaller Qwen3.5-0.8B model.

I trained the 0.8B setup with limited resources, mostly using free Kaggle notebook GPUs.

The setup keeps both the **Qwen3.5-0.8B backbone** and the roughly **51B-parameter PLE memory** frozen. A small R=1 reader is trained at decoder layers 3 and 9, with a token-dependent linear gate controlling the later injection.

There is **no backbone fine-tuning**.

The current balanced checkpoint uses a reader trained for 15M tokens and a separately calibrated dynamic gate. On the frozen full-validation set:

**Qwen3.5-0.8B stock**

* NLL: 2.905585
* PPL: 18.2759

**Qwengram-0.8B**

* NLL: 2.853786
* PPL: 17.3534

**Perplexity reduction: 5.05%**

This is a language-model validation result, not a claim of 5% higher benchmark accuracy.

A few findings shaped the final design:

* The real pretrained PLE outperformed both random-memory and permuted-memory controls.
* Reader loss kept improving well beyond 5M training tokens. The 20M reader improved aggregate LM loss further, but regressed on math, so 15M remains the balanced checkpoint.
* Strong fixed late-layer memory injection hurt LAMBADA. Dynamic token-level arbitration recovered much of that tradeoff.
* The gate is genuinely dynamic: its memory strength varies substantially across tokens rather than behaving like a learned constant.
* With the exact same memory budget, learned token placement beat shuffled placement. Routing memory toward high-uncertainty positions recovered part of the advantage, but still did not match the learned gate.
* A warm-started R=4 reader produced a small aggregate LM-loss improvement, but introduced code and math regressions. I therefore kept R=1 as the balanced architecture.

I also implemented the inference path in llama.cpp.

The public artifacts are:

**Model / GGUFs**  
[https://huggingface.co/Ninnix96/Qwengram-0.8B](https://huggingface.co/Ninnix96/Qwengram-0.8B)

**Training, controls, and evaluation**  
[https://github.com/Ninnix/qwen-ple-transfer](https://github.com/Ninnix/qwen-ple-transfer)

**Modified llama.cpp runtime**  
[https://github.com/Ninnix/llama.cpp](https://github.com/Ninnix/llama.cpp)

The GGUF contains the Qwen3.5 backbone plus the trained reader and arbitration tensors. The large PLE remains an **external quantized sidecar**, rather than being packed into the model GGUF.

I also tested quantization retention on a separate fixed WikiText-2 GGUF runtime test:

* **Q8\_0 retains 99.1% of the BF16 reader NLL gain**

This is a separate runtime measurement, not the frozen Kaggle validation benchmark above.

I’d welcome attempts to reproduce or improve the reader, PLE caching, routing, or runtime.

Next I’d like to try larger Qwen backbones, particularly the **35B-A3B MoE**. Experiments at that scale require substantially more compute than free Kaggle notebooks can provide, but the 0.8B study gives a much clearer recipe for reader scaling and dynamic memory arbitration.

>**Disclosure:** I’m the author of Qwengram and the linked repositories. English is not my first language, so I used AI to help proofread grammar and improve phrasing in this post. The experiment itself was also developed with the assistance of coding agents, primarily ChatGPT Sol, for implementation, debugging, experiment orchestration, and analysis support. I designed the experiments, made the research decisions, reviewed the results, and am responsible for the final conclusions.

document fetched 2026-09-26 00:41 · canonical https://reddit.com/r/LocalLLaMA/comments/1wpvep4/qwengram08b_i_transferred_qwen38_flashnexts_ngram