alt Hacker NewsWell, what’s the theorem statement for the result you’re thinking of then?
Of course n bits can only encode for 2^n options. But to prove mathematically that we are close to the best that can be achieved in 1GB requires a mathematical definition of what we mean by better. Rather, it requires at least a choice of a proxy for what we mean by better. (Showing that some approximation of what we mean by better is close to as good as can be, would suffice. Like, if we can show that the loss can’t get much lower with a 1GB model, that would count.)
We generally don't care enough to test it since it would be a significate waste of resources that's why we have models in the trillions of parameters instead of continuing to train smaller models smaller models hit a wall and generally stop improving and start overfitting and generalization starts to degrade it's less of a theorem, but an observation that has occured during training and documented here and there across many papers.