alt Hacker NewsThe theorem you want to pay attention to is the no free lunch theorem. The important thing to understand there is that the larger models give you "free lunch" in the sense that you can approximate more different systems accurately at the cost of model size. If there was a Karpathy style universal solver, it wouldn't be very smart unless we scaled it up.
This isn't to say that there aren't a fair amount of wasted parameters in current LLMs, but then we already kinda knew that since you can quantize models down to 3-4 bits per weight often times with minimal loss.
> If there was a Karpathy style universal solver, it wouldn't be very smart unless we scaled it up.
I think that the scaled up version is actually still really valuable.
Imagine being able to just add more compute as needed for any given problem until it's solved by just adding more copies of a single universal layer, without more training. Or being able to burn the individual core into silicon and just loop it as needed.
I tried to build exactly that in my personal lab once, but hit a wall made of my own incompetence and budget.
The idea was to find the parts of the manifold that did generic reasoning and then scale as needed by repeating them. It worked within individual layers (I could make the model score higher on benchmarks by repeating the reasoning extracts within individual layers), but i could never get the interfaces between layers to work again after I'd done that. I suppose it needed traing to "heal" the interface again after my brain surgery, but I didnt have the compute to manage it and moved on to the next project
I'm sure that someone who actually gets paid to do these things will figure out some version of it eventually though, because I know it can be done.