alt Hacker NewsWhat I can not wrap my head around: How are these models trained?
What training mechanism or model architecture provides the glue to go from human text to images?
Don't you need to have millions of really descriptively labelled images?
Replies
vonneumannstan • today at 2:54 PM
Short answer yes.
Slightly longer answer for older text to image models you teach them how to encode images and text into the same latent space. Then you simply do a conversion, take a text input, put it into latent space and then extract the image that latent space represents.
mistercheph • today at 2:33 PM
I found this 3b1b guest video on diffusion helpful: https://www.youtube.com/watch?v=iv-5mZ_9CPY
That's exactly how they do it.
There are ML models that do the reverse and output image to text, which assist quite a lot.
The better the text represents the unique thing in the photo, the better the model understands what that text means.