alt Hacker NewsTo make a rather speculative argument - since an LLM should be able to describer an image losslessly, images are a subset of what an LLM needs to understand. It is possible that the statistical model being learned for images is just simpler.
The get a proper LLM they have to model all of human knowledge regarding patterns, physics, biology, social interaction, skills, history, etc, etc. For images they have to recall what a dog looks like in a still photo. The image models aren't even necessarily good at a wide range of novel situations; usually they get used for stock scenes.
Natural evolution has also had a much easier time coming up with vision models (everything with eyeballs) than complex language models (as far as we know only a smallish family of monkeys).
Replies
> since an LLM should be able to describer [sic] an image losslessly
Any lossless text description of a given image would just be a structured text output of the exact pixel values, or an encoding that losslessly exploits regularities. Anything less is categorically lossy. An ordinary semantic description of a reasonably complex image will generally be lossy. A short, lossless text description would need to exploit regularities in the image, such as a solid-color image, or a basic pattern or sequence which is easily reproducible, such as a precisely specified checkerboard image.
And being able to represent images losslessly in text simply establishes text’s capacity to encode information. It doesn't establish an LLM’s holistic understanding of images. There is heavy lifting in the phrase “images are a subset.” An image can be almost any conceivable perceptual arrangement of data. The bounds are not clear. Maybe what you meant to suggest was that images of our natural world contain exploitable regularities.
> Natural evolution has also had a much easier time coming up with vision models (everything with eyeballs) than complex language models (as far as we know only a smallish family of monkeys).
It isn't appropriate to explain complex biological systems in terms of today's technologies. This is something people have been doing for a long time. The mind is like a a book, a loom, a camera, an engine, a computer, an LLM/vision model. Etc. The cognitive abilities required for an organism to meaningfully interpret its vision can possibly largely be the same cognitive abilities underlying speech, and all that is missing for a brain to exploit this might be sufficiently advanced vocal cords or manual dexterity; nothing to do with the brain itself, but its auxiliary organs.
this is the right answer. ultimately text and general llm usage incl coding is far more internally complex than just memorizing the looks and composition of concepts placed in an image. dont forget that most of our reasoning benchmarks basically require the llm to have a search engine and calculator in its own weights.