alt Hacker NewsIf you know how an LLM could have an output image head, that is purely generative without using diffusion, and not a classification/selection head, I’m really interested in learning more. I think you are onto something very new and novel here
If you know how an LLM could have an output image head, that is purely generative without using diffusion, and not a classification/selection head, I’m really interested in learning more. I think you are onto something very new and novel here
Well I doubt that :-) the image tokens can just be added as further tokens, so one wouldn't even need a dedicated output head. E.g. Emu architecture [1] does text, image, and videos all together. So definitely not new. That's not even a separate head, everything stays together.
Regarding whether that's standard to pre-train multi-modal LLM, apparently that is not the case (at least, according to ChatGPT's answer, but that's reasonable - or anyway it was only a wild guess on my side). Apparently it's more common to have a text-only and an image-only pretraining phases, followed by an alignment to make the two embedding spaces a common one.
[1] https://arxiv.org/abs/2409.18869