alt Hacker Newsthis is the right answer. ultimately text and general llm usage incl coding is far more internally complex than just memorizing the looks and composition of concepts placed in an image. dont forget that most of our reasoning benchmarks basically require the llm to have a search engine and calculator in its own weights.
These diffusion models are not just memorizing the looks and composition of concepts. These models are increasingly capable of representing advanced world knowledge. Things like shadows require a comprehensive, if indirect, understanding of physical light modeling. Things like weight/size estimation, etc., which are all important to a cohesive generated image of natural phenomenon, require an adequate understanding of physics.
Some of this understanding is in the input embeddings themselves, embedding models do a lot of heavy lifting behind image generation models.
I posit that you can theoretically achieve the same capability in a model trained on the right sequence of "images" as you can purely text; both are turned into embeddings and can ultimately convey the same exact relationships. I don't think it's very useful to try and create delineations between text and image inference, especially as more models move towards native multimodality.