alt Hacker NewsHasn't it been repeatedly shown that many small models perform worse than a large model of the same total parameters?
Replies
ilaksh • yesterday at 10:46 PM
This is not a normal small model. It's in a different category of tiny.
Hasn't it been repeatedly shown that many small models perform worse than a large model of the same total parameters?
This is not a normal small model. It's in a different category of tiny.
I think that is a sort of a reverse scaling fallacy. Given the right resources and environments, many small models can function together in an emergent way. I’ve been on the lookout for an SLM version of Conway’s game of life. SLM always reminds me of slime molds, which demonstrate a form of intelligence which is remarkable.