alt Hacker NewsI could never get this about modern machine/deep learning or even the Transformers. Yes, it's not exactly rocket science, but when I see the data flow diagrams, it's not clear what is calculated in real time or multiple steps.
Is it really one big computation f(g(h(x)))?
Replies
choilive • today at 5:08 PM
What's your distinction between real time vs multiple steps? All computation is done in steps.
Is it all one big computation? Its turtles all the way down.
leonvoss • today at 4:57 PM
It's all vibes.
Yes.
Each token prediction is one big function call. Then you just recursively generate more tokens until run out of context or the model predicts a next token indicating end of sequence. Technically the model outputs a matrix where the last row is a probability distribution, but I’m counting sampling from it as part of the chain. Hundreds of billions of dollars has gone into just making the function fatter and gradually changing pieces here and there.