Just skimmed the paper (haven't seen this before). Might need to read more carefully, but at first glance I don't understand their intuition why the "Markovian property limits the model’s ability to fully utilize the generation trajectory".
When it comes to multi-resolution training (e.g. matryoshka training), there are precedents that don't require this AR formulation.
They cite MAR (https://arxiv.org/pdf/2406.11838), which I think is a much clearer articulation of "autoregressive diffusion". MAR uses an autoregressive base (like an LLM) and staples on a small MLP on-top that's trained as a diffusion head. That makes more sense to me, since you can leverage the "knowledge prior" from an LLM and have it generate images. That's probably how nano-bannana and GPT-Image broadly work.
Zooming out, it's not clear to me from any of the work WHY autoregressive diffusion on it's own is better than regular diffusion. Most autoregressive diffusion models use speculative decoding, because pure autoregressive diffusion is too slow to run at inference time.
The one ~magical~ thing about auto-regressive diffusion (IMO) has nothing to do with autoregressive vs. fully-bidirectional diffusion. But simply, the face that you can do it on-top of a LLM. That means the LLM can look at it's generation, assess it's quality, think on what's broken, and then call itself to edit the image and fix it. With methods like RLVR, that means you can essentially guarantee the correctness of your image (along the axes you've RLVR-ed).
alt Hacker News
Just skimmed the paper (haven't seen this before). Might need to read more carefully, but at first glance I don't understand their intuition why the "Markovian property limits the model’s ability to fully utilize the generation trajectory".
When it comes to multi-resolution training (e.g. matryoshka training), there are precedents that don't require this AR formulation.
They cite MAR (https://arxiv.org/pdf/2406.11838), which I think is a much clearer articulation of "autoregressive diffusion". MAR uses an autoregressive base (like an LLM) and staples on a small MLP on-top that's trained as a diffusion head. That makes more sense to me, since you can leverage the "knowledge prior" from an LLM and have it generate images. That's probably how nano-bannana and GPT-Image broadly work.
Zooming out, it's not clear to me from any of the work WHY autoregressive diffusion on it's own is better than regular diffusion. Most autoregressive diffusion models use speculative decoding, because pure autoregressive diffusion is too slow to run at inference time.
The one ~magical~ thing about auto-regressive diffusion (IMO) has nothing to do with autoregressive vs. fully-bidirectional diffusion. But simply, the face that you can do it on-top of a LLM. That means the LLM can look at it's generation, assess it's quality, think on what's broken, and then call itself to edit the image and fix it. With methods like RLVR, that means you can essentially guarantee the correctness of your image (along the axes you've RLVR-ed).