alt Hacker NewsYes.
Each token prediction is one big function call. Then you just recursively generate more tokens until run out of context or the model predicts a next token indicating end of sequence. Technically the model outputs a matrix where the last row is a probability distribution, but I’m counting sampling from it as part of the chain. Hundreds of billions of dollars has gone into just making the function fatter and gradually changing pieces here and there.
I remember the concept of layers, as essentially defining the matrix math dimensions. And for a given model/framework, they were static. That always bugged me (not very dynamic).. is this still the case?