alt Hacker NewsIt is infact mathematical proof that is somewhat well understood although specifics are debated. We do in fact know that the limit follows information density law which is something we can measure and we are pretty close to it.
It is infact mathematical proof that is somewhat well understood although specifics are debated. We do in fact know that the limit follows information density law which is something we can measure and we are pretty close to it.
Well, what’s the theorem statement for the result you’re thinking of then?
Of course n bits can only encode for 2^n options. But to prove mathematically that we are close to the best that can be achieved in 1GB requires a mathematical definition of what we mean by better. Rather, it requires at least a choice of a proxy for what we mean by better. (Showing that some approximation of what we mean by better is close to as good as can be, would suffice. Like, if we can show that the loss can’t get much lower with a 1GB model, that would count.)