alt Hacker News> They're likely to not be feasibly scalable far beyond the 26B size of DiffusionGemma itself
I think people used to say the same about the 8B text-diffusion models too when they came out, like LLaDA. LLaDA2.0 seemingly claims 100B total / 6.1B active MoE diffusion (DiffusionGemma is also MoE). Not saying you're wrong about the current consensus, but it has a way of changing over time, might be a bit early to claim it's infeasible to scale them, especially considering the final artifact being much more suitable for local usage.
Difficulty of scaling is not the only issue. Nobody is going to be particularly invested in scaling an architecture that has:
- consistently proven behind their auto-regressive counterparts in quality. Look at the dgemma benchmarks - pretty steep dropoffs and the more difficult the benchmark the worse the dropoff. That's not a good look and it's not like its some artifact of google's release. Every dllm is like this.
- And whose inference benefits are negated at scale. Transformers are still cheaper if you want to serve lots of users.
>"DiffusionGemma's speedup is designed for local and low-concurrency inference. In high-QPS cloud serving, autoregressive models can be deployed to saturate compute efficiently, so DiffusionGemma's parallel decoding offers diminishing returns and can result in higher serving costs"
Put yourself in the shoes of all the labs, even open source ones. Why would you put much effort into this ?