alt Hacker NewsI worry this is looking at where the ball is now instead of where it's going. The recent disproof of an Erdos conjecture should put to rest the idea that LLMs will reach a skill ceiling before they reach superintelligence.
I believe we are headed for a world of superintelligent AI where LLMs are much better at logical thinking than humans, the same way that chess engines are much better at chess than humans.
In that world there's really nothing humans can offer in terms of logical thinking other than their humanity itself. An 8 year old with Stockfish can beat Magnus Carlsen, and an 8 year old with Codex (and daddy's credit card) will be able to beat me at software engineering.
Replies
I personally don't see that much sense worrying about this scenario because if it comes true then it doesn't really matter what I do.
Are you confident in putting a timeline on this prediction?
One of the reasons I'm increasingly skeptical of this prediction is that I've now lived past a few of the dates I heard people put on the achievement of this level of superintelligence in previous years.
If building software (and even programming, as the basis for it) was just an expression of logical thinking, we would have cornered it long time ago IMO.
But then again, logic is really a lot more discrete and well defined and easily expressed with traditional computing than LLMs are (which are probabilistic systems instead and as such require large knowledge bases).
We can observe that at a couple hundred billion parameters they behave similarly to a point (in the sense that they can produce similar results), but the challenge is really in understanding the problem's multifaceted structure and competing needs and priorities.
Chess and proofs only work as comparisons to the extent that you can find parts of your job that share their key property: A solution is sought to a problem that can be stated with relatively little information.
What prompt would someone have used to get a superhuman coding agent to output the Linux kernel or GTA5?
Before you accuse me of moving the goalposts, that's not my point: The examples are there to help think about what humans would still need to do to build complex projects even if the coding itself was perfectly reliable.
Both the Linux kernel and GTA5 contain a large amount of incompressible information; humans thought long and hard about how to design them, i.e. about what that thing they were building was even supposed to be.
> In that world there's really nothing humans can offer in terms of logical thinking other than their humanity itself.
By your logic anyone who's not in the top 10% of intelligence can't offer anything. The world keeps spinning.
> An 8 year old with Stockfish can beat Magnus Carlsen, and an 8 year old with Codex (and daddy's credit card) will be able to beat me at software engineering.
That's just nonsense, nobody will work with 8 year old (it's illegal, to start with). Go touch grass.
I don't buy that at all.
It doesn't matter how great the LLMs get, the act of creating software using them will still require a great deal of skill.
Most people just don't think in terms of software.
Try asking a non-developer in your life what their dream software would be for their work, or their hobby. If they don't have what Nilay Patel calls "software brain" I'd be surprised if they came up with something actionable.
(For more on software brain see "THE PEOPLE DO NOT YEARN FOR AUTOMATION", which makes the point I"m making here but much, much better: https://www.theverge.com/podcast/917029/software-brain-ai-ba...)
You could give a non-developer the smartest LLM in the world and they wouldn't be able to create GitHub with it, because creating GitHub requires an enormous amount of understanding of what software developers need from a cloud source control tool.
Sure, you can argue that the LLM "knows" what GitHub needs already and can guide their human-user to that, but why would a human-user who doesn't understand the domain ask an LLM to do that in the first place?