When Atari games or Go were solved around 10 years ago, nobody was saying that superintelligence was within months just because reinforcement learning worked on a domain that allowed learning by self-play. Now we have a domain that is solvable by reinforcement learning with verifiable rewards. It's very impressive that these models are capable of doing the things they do, but the techniques aren't transferable to the vast majority of other domains. ML research also isn't verifiable in the sense that math is. People who claim it's verifiable, because there's a target that can directly be optimized don't understand why RL works for math. It's that lean is able to massively shrink the search space. This simply cannot be done in other domains in the same way.
I'm not a fan of Dwarkesh, but even he laid out that machine learning research progress is very much verifiable. At least in the RSI sense, that our current approach can be drastically optimized.
The models are improving at certain fields, and one of those fields is creating new models, which are verified by applying them to fields where you can prove that they're better.
I agree they don't transfer to all domains, but we do not know how far the theoretical will overlap with practical science. It could be "not all that much" and we hit roadblocks quite soon, or it could be that as the models become better and better at reasoning and research within their theoretical domains, it turns out to translate really well to the real world.
We don't know, but my bet is that we will continue to see rapid progress and impacts on the real world, until we reach a point where sceptics will have to throw in the towel.
LLMs don't produce Lean proofs directly, they produce the proofs in natural language, and only translate them afterward. The fact that we have Lean has nothing to do with LLMs being good at theorem proving.
What kind of a thing do you think AI won't be able to do in 2030? A specific thing please.
Yeah, a lot of the RSI claims seem dependent on us having very specific knowledge about how much intelligence is required to build superintelligence that I haven't seen anyone claim to have. Maybe you have to be, I don't know, 100x smarter than humans before the recursive part of RSI becomes tractable. Or maybe there are fundamental limits you can do with the hardware that exists that we don't know about, which means building superintelligence will take more time. I'm not saying the 2027-2030 people are wrong, I just think it depends on a bunch of things we can't credibly claim to know at this point.