alt Hacker News > But our models make it clear that such an [intelligence] explosion may not follow if there are diminishing returns (“ideas become harder to find”) or if feedback loops become bottlenecked.
Replies
Because they depend on whether the rate of improvement of self-improvement outpaces the rate of increase in difficulty or not, and at some points they clearly do - e.g. a lot of skills makes the relative rate of subsequent improvement easier for a while.
It may seem obvious that it can't last, but showing the conditions where it can't still matters.
> As we advance it becomes more difficult to advance
I don't think this follows.
We have advanced tremendously over the past 200 years, and we are likely going into a time with rapid advancement again.
With advancement, we also develop tools (eg. Llms) that assist advancing.
> As we advance it becomes more difficult to advance. You obviously make most advancements around the things that are easiest to improve. Then all the easy things are done.
This isn't some foregone conclusion. It completely depends on the rate at which the intelligence and abilities of the AI increases. If that rate was high enough, then the harder and harder problems would become easier and easier for it.
>How is this not obvious to everyone?
because people have other ideas https://en.wikipedia.org/wiki/Technological_singularity
> How is this not obvious to everyone?
That RSI can be bottlenecked? I guess this is obvious to many people. Whether RSI will be bottlenecked (at some not very interesting stage) is another question.
Progress is often lumpy, modulated by new discoveries and their applications.
"How is this not obvious to everyone? As we advance it becomes more difficult to advance. "
It's a type of recent change blindness. Because for the last few years there are seemingly impossible breakthroughs every few months. Literally, things everyone said would take 30 or 100 years, happen within months. It is really easy to think this will continue. The new normal. Really, we haven't seen it slow down at all, so why would we expect the miracles to stop all of a sudden?
It seems like accelerating, and decelerating, are both fair game as future directions at this point.
So it is nice that the paper put some weight behind the argument that this could all grind to a halt for a lot of different reasons beyond the hype.
> How is this not obvious to everyone?
Probably because it's not true. We had shitty neural networks for decades before the recent explosion. That particular branch may be a dead end, but there could be others lurking and waiting for their time.
> How is this not obvious to everyone?
Because everyone's thinking around intelligence is incredibly muddled by a variety of factors, and no one is particularly motivated to actually correct anyone's mistaken notions on the matter.
The goal of a modeling exercise like this, which you don’t have to buy, is to generate a simple set of initial conditions that can explain things we already know. Then, we can manipulate some initial parameter value to make predictions about things we don’t see, but might.
Likewise, it is obvious that gravity exists, but a simple model that explains where it comes from (in quantum terms) would be a big breakthrough iff it came with plausibly testable implications that could be tested via experiment.