# I trained DiffusionGemma to be an abstract expressionist — retained record

> Retained reddit record from reddit:LocalLLaMA.

Canonical: https://highsignal.app/data/records/d858e4f25f55246e

[← sources](https://highsignal.app/data)


source: [reddit](https://highsignal.app/data/reddit) (reddit:LocalLLaMA)  published 2026-11-13 11:48 ingested 2026-11-14 02:42

[original source ↗](https://www.reddit.com/r/LocalLLaMA/comments/1xv5svo/i_trained_diffusiongemma_to_be_an_abstract/)

## retained text



```
  
*DiffusionGemma, a text diffusion model built on a Gemma 4 backbone, can be adapted into a shitty image generation model.*

[Maybe the ship sank?](https://preview.redd.it/e0rvo28r6d1i1.png?width=840&format=png&auto=webp&s=cbe7e768c7522207a308e0862e5ea8d975a08ada)

You may be asking yourself, "WTF, why would you want to do that?" And you would be right to ask. This is honestly kinda useless, but I still think it's [pretty neat](https://www.youtube.com/watch?v=Hm3JodBR-vs&t=58s). Also, fair warning, being on a team of one means the project has been heavily — I'll be nice to myself and say — LLM *assisted*. I invite you to sidle on up to the trough and dig in to the slop!

Still with me? Good, because I've been poking at this for a while, on-and-off since DiffusionGemma was released back in June. I was immediately hooked on this model as soon as I read [its release blog](https://blog.google/innovation-and-ai/technology/developers-tools/diffusion-gemma-faster-text-generation/) and [model card](https://ai.google.dev/gemma/docs/diffusiongemma/model_card). 4x faster text generation they said — hell yeah! But here's the thing: **this model sucks.** I mean it's a great chat bot. It can do simple NLP tasks, and it's really quick for text generation as they say, especially for long-form output. But when it comes to agentic tasks and reasoning it's a total loser.

* 0% on ARC-3 (but that's not really a fair dig, this is a tough one)
* 0% on ARC-2
* 0% on The Gemma 4 Developer Agent Competition (this competition targets the base model DiffusionGemma was trained from)
* 11% on HLE on the model card (9.2 measured in deployment)
* 18.7% on SWE-bench Verified

So not great, and particularly bad when you consider that on the hardware you'd need to run a good quant of DG (so basically bf16), you could easily run qwen-3.8-27B or a respectable FN quant.

But please don't get me wrong. **I absolutely love this model.** Or at least I love monkeying around with it, and I doubt I'm the only one. Huge shoutout to the Unsloth team BTW! Reading their [Colab notebook training DG to play sudoku](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/DiffusionGemma_(26B-A4B)-Sudoku.ipynb) is what made me start asking the questions that led to this project.

Alright, enough fluff. So what did I actually do. Well the thing I'd headline is that **I created a text to image and image manipulation training and inference pipeline for DiffusionGemma and then used that pipeline to adapt DG (like <300M parameters total) to generate things that are recognizably images and to perform visual operations on images that are provided to it.** Surprisingly, it took only a very small amount of data and compute to start getting interpretable results.

*But DiffusionGemma did not want to do this.* The thing that finally worked consistently was prefilling the prompt all the way to the start of an input string for an "image generation tool" described to DG in the system prompt as a tool that it could use to convert text strings into images, but DiffusionGemma itself was actually the one doing the generation, not an external tool.

[Here's the flow.](https://preview.redd.it/37xv8dtw6d1i1.png?width=2400&format=png&auto=webp&s=2dae87f4c4406234cb40bae80f86020ca5d16d21)

*This also required patching an* [*upstream bug*](https://github.com/huggingface/transformers/pull/50684) *in Transformers for image attention, so take note if you wanna try this out for yourself: you'll need to patch until PR is merged.*

Considering how bad DiffusionGemma is at everything else, I'd say it does a pretty good job here. 

[It's like human centipede!](https://preview.redd.it/1ckymu0m7d1i1.png?width=1980&format=png&auto=webp&s=1e89062873d0df9dd9219cfac2845e0742d65654)

But wait, there's more! Since DG is outputting in its native visual latent space (one set of 256 tokens at a time through a trained output head) it can take the visual representation it is generating as input for the next forward pass — on-domain block autoregressive, just like with its regular text canvases. And not just can it do this, but it actually helps improve quality and generalization (on image manipulation tasks)!

[Call graphs, the first hint of a non-stupid use case.](https://preview.redd.it/atc8jwyw7d1i1.png?width=2400&format=png&auto=webp&s=e6ec1996726b3f725e6b90794f8a2fbfd06e9b64)



**Here's a super-quick breakdown of the pipeline.**

The first thing that had to be done was to train an actual vision *decoder.* This turned out to be the easiest part of the entire project. One evening, on a whim, I threw a prompt at my local model (dsv4-flash-vision-exp I believe was what I was running then), something like: "D'oh! Could we train a decoder for the ViT soft tokens that works for 768^(2) images?" And then it did it. Took like a few hours, and works pretty damned good I think.

[It decodes!](https://preview.redd.it/h2vdpvo58d1i1.png?width=2400&format=png&auto=webp&s=787234c2fc6392af2bd35d2f8971e60d98027561)

Alright so that let me nail down the decoding and supervise the pixel reconstruction loss, but let's not forget DG is a **text** diffusion model — *it has to actually operate on a text canvas*. That means I needed to derive a text target image representation (or codebook) that was precise enough it could be used for supervision without over penalizing the model for expressiveness.

This process itself ended up being a bit like a diffusion operation. I simply threw up a wall of half-baked, ill-advised, and just plain stupid ideas and worked on them all until I was confident in each — with that confidence being mostly confidence in them not working. Then I had the brilliant idea: just ask DiffusinoGemma itself for the target.

This essentially consisted of me badgering DG until it would describe images as text. Okay, not really, but that is kinda what it felt like. The basic idea is: encode an image and ask the model to describe each soft token patch in a single token (word, number, symbol, unicode, CJK character, etc). Each soft token having come through the ViT is contextualized against the full image, so this description would most often carry a structural/semantic channel. To get the photometric channel you do basically the same, but instead of a full image use only a single (48\*48 pixel) patch. This usually carried the color/texture/photometric channel. Sample the logits for each text description token and then it's basically chef's choice on how to combine across all prompt and patch variants into a workable target.

Doing that for every image in the training set isn't reasonable. IT'S OVER 9000! However, doing exactly that for a small set of images (200ish) and using that as a training set for a teacher-forced model that then generates the full dataset turned out to be a very smooth and successful approach.

Anyway, the point is that the combined multi-modal target token distributions for the canvas text results in training DiffusionGemma to be an S-tier emojipasta generator. 

[Example training targets](https://preview.redd.it/8hj4uv1c8d1i1.png?width=1560&format=png&auto=webp&s=8873012515562949f0f436c129c1b7a795793c8f)

[DG's generated canvases.](https://preview.redd.it/gnhts2mh8d1i1.png?width=1600&format=png&auto=webp&s=5b5c609ce54461598bc270410bae9e454cdffae7)

Then I LoRA adapt DG and simultaneously train a new image output head that feeds into the previously trained image decoder supervised against supervised against the target token set, ViT latents and, decoded image. Basically training DiffusionGemma to generate emojipasta that decodes into image-like objects. I guess this would be something like a VQ-VAE, but using text tokens instead of dedicated image vectors because stock DG can read and understand the image latents, but can't natively generate them, only text.

Well, that's the gist. **Thanks for reading!** If you find this as interesting as I do and want to learn more or play around with this yourself you can:

* [Access the adapter and decoder weights on HF](https://huggingface.co/bumblebeer/dgg2-latent-code-graphs-adapters)
* [Grab the training and inference code from GH](https://github.com/bumblebeers/dgg-latent)
* [Or read more in depth, and about how this could potentially be applied to agentic coding tasks on my Kaggle writeup](https://kaggle.com/competitions/gemma-4-developer-agent-paper/writeups/new-writeup-1794372276727)

**tl;dr: Please GDM team give me a stronger text diffusion model to play with!**
```



document fetched 2026-11-14 00:46 · canonical https://reddit.com/r/LocalLLaMA/comments/1xv5svo/i_trained_diffusiongemma_to_be_an_abstract
