alt Hacker NewsThese algorithms don't have intelligence, they just regurgitate human intelligence that was in their training data. That also goes the other way - they can't produce intelligence that wasn't represented in their training input.
Replies
Firstly, it doesn't really matter if they can produce novel designs or not. 99% of what is being done is not novel. It is manipulating data in ways computers have been manipulating data for decades. The design of what is implemented is also going to be derivative of what already exists in the world too. Being too novel makes for a bad product since users will not easily understand how to use it and adapt their existing knowledge of how other things work.
Secondly, they are able to produce intelligence that wasn't represented in their training input. As a simple example take a look at chess AI. The top chess engines have more intelligence over the game of chess than the top humans. They have surpassed humans understanding of chess. Similar with LLMs. They train on synthetic data that other LLMs have made and are able to find ways to get better and better on their own. Humans learn off the knowledge of other humans and it compounds. The same thing applies to AI. It is able to generated information and try things and then later reference what it tried when doing something else.
How does post-training via reinforcement learning factor in? Does every evaluated judgement count as 'the training data' ?
"regurgitate human intelligence from training data" - exactly. And the tricky part is when your actual code contradicts that training data.
Model saw thousands of examples of "how to implement X". When your codebase does X differently, training data wins. You can see it happen: point out the conflict, and a model that's reasoning would shift gears - ask questions, acknowledge tension. But model in retrieval mode just reiterates. Same confidence, same explanation, maybe rephrased.
That's why "I completely understand this time" keeps happening in AI responses. From model's view, nothing to check - the pattern it retrieved already "makes sense."
In short, if you're not doing something completely new, something that AI will almost certainly do better than you, then you're safe. Otherwise, you'll have to put in a considerable amount of effort to get the AI to cooperate, or you'll have to do the most difficult tasks yourself, simply because the limitations described above haven't been addressed.