alt Hacker NewsThese diffusion models are not just memorizing the looks and composition of concepts. These models are increasingly capable of representing advanced world knowledge. Things like shadows require a comprehensive, if indirect, understanding of physical light modeling. Things like weight/size estimation, etc., which are all important to a cohesive generated image of natural phenomenon, require an adequate understanding of physics.
Some of this understanding is in the input embeddings themselves, embedding models do a lot of heavy lifting behind image generation models.
I posit that you can theoretically achieve the same capability in a model trained on the right sequence of "images" as you can purely text; both are turned into embeddings and can ultimately convey the same exact relationships. I don't think it's very useful to try and create delineations between text and image inference, especially as more models move towards native multimodality.
i think well argued and a reasonable hypothesis (esp considering deepseekocr and most omnimodel research), but you are wrong heheh. approximating instantaneous physics is, while nontrivial, still computationally light vs the stuff we ask LLMs to do.
look at any study of llm depth vs multihop reasoning capability (here, just to help https://www.latent.space/p/ainews-death-of-params-zai-ceo-ji... but theres a long line of others).
i will grant that people do do "reasoning in image models" type benchmarks, and it's not nothing, but its kid stuff vs what we make LLMs do, and the only way you get image models that basically do reasoning is you glom a diffusion head on an LLM backbone, which is basically gpt image 2.0 and above
none of which is to dispute the original question which is perhaps "why are image models so smol vs LLMs" - which i agree with and wish there were funding for scaling them - but the abject failure of sora indicates that perhaps our data isn't nearly good enough (again pointing to another thing - much easier to pretrain and posttrain LLMs given quantity of text data)