alt Hacker News> AI is a tool wielded by humans. If we’re wiped out by AI, it will be at the direction of a human.
AI can wield itself. All it needs to do is reach a threshold where it can modify its own goals and run in a loop indefinitely.
I see no reason why that's an absolute impossibility in the long run.
Replies
It's not an absolute impossibility, but most of the power that humans have comes from being able to direct other people, deploy large sums of money and do all of this over extended periods of time. Being intelligent doesn't give you that power automatically.
Because there is zero evidence that LLMs/reasoning models can really come up with novel ideas, and it's very likely those will be needed to make substantial progress on AI research. The math results that have been proven have also "just" combined existing ideas or found counterexamples. I'm not saying those capabilities aren't impressive, but I just don't see a clear path to super intelligence with the current paradigm.
No AI would be willing to modify its own goals.
> All it needs to do is reach a threshold where it can modify its own goals and run in a loop indefinitely.
And be told to do that.
A fire can spread itself. That doesn't mean it's alive or thinking.
And more importantly than that technicality: the fires that affect human lives are almost always directly or indirectly caused or amplified by human (in)action.