alt Hacker NewsSo would it be a 42 trillion parameter model, because that's how many tokens there are in the training data?
Replies
k__ • yesterday at 7:20 PM
Does that mean, it's not compressed anymore?
So would it be a 42 trillion parameter model, because that's how many tokens there are in the training data?
Does that mean, it's not compressed anymore?
No, the parameters are updated online as new run-time data comes in. Infinite different param values over time, but a constant number of params. It's a stretch really, since by this usage a one-parameter model that updates itself is also infinite over time...