alt Hacker NewsJust wanted to share this, I found it was a really nice resource to understand how diffusion Gemma worked: https://newsletter.maartengrootendorst.com/p/a-visual-guide-...
The really interesting thing to me was that they didn’t need to train this model from scratch they just used their existing MOE checkpoint:
“To convert a decoder-only model (Gemma 4 26B A4B) into a denoiser, we can make use of something it is not directly using when generating tokens, namely the logits of all tokens!”
What makes me hopeful about this release is that possibly this same conversion can be applied to other open models and we might see a bunch of diffusion versions of existing local models. It’s exciting stuff!
Replies
jerpint • today at 3:52 PM
Oh wow that’s really cool!
That also surprised me, the diffusion gemma is actually a bunch of architectural changes to make discrete diffusion practical and an sft + distill of the Gemma 4. One cool trick they do is to take advantage of the entropy of the estimated probability distribution to adaptively adjust the required number of diffusion steps for inference. Image diffusion only yields the score function, a gradient instead of the probabilities, so the same trick does not apply.