alt Hacker NewsI think this is an outcome of playing with systems that we don’t actually understand very well on a deep level. We have found recipes that work, and have used graduate descent (so many PhD theses) to understand and optimize these systems, but we’ve built a model of language that also encodes reasoning, and those things are extraordinarily complex. Providing proper mathematical rigor for their explanation requires, as you say, the level of commitment equivalent to a math degree. Also, because language is so complicated and subtle, you need highly precise statistical models of it, and that just takes a shitload of data; hence the scale. You can train toy models at home, but they will suck because you actually need massive amounts of data to capture the nuance that language requires. And that’s to say nothing of multimodal models.
Replies
You say "...because you actually need massive amounts of data to capture the nuance that language requires..." but humans don't require that! So why do LLMs?
How many books did you read in your education? Far fewer that what LLMs read, no doubt. Of course we "have a life" and so we interact with hundreds of people, which is also a form of learning.
But in any case there's are fundamental differences here between these two systems (LLMs and humans) that we have not yet discerned clearly.
I think the level of interaction on forums like HN is a great example of the still-yawning gulf between LLMs and humans when it comes to encoding reasoning into language. Even with the self-selection that comes with posting here, think of how many times it is almost impossible to tell if posters are shit-posting, serious, angry, sock puppets, or just weird.