alt Hacker NewsAssuming it keeps improving at the same rate, which I think we are already seeing not play out. If you compare the first six months when GPT truly hit the mainstream to the previous six months, the improvements are not nearly as evident. That isn’t to say they aren’t noticeable, I could definitely tell it’s improving, but not nearly at the pace it once was.
There’s also the fact that they can’t possibly keep improving frontier models at the same rate (I.e. training investment) when investment starts slowing down. The amount of cash being burned is completely unsustainable and you’re already seeing some pullback.
Replies
On the other hand we keep seeing only marginal generational imorovements in CPU space, yet performance gains over last 10 years in CPUs are very material.
Every new model might not be a leap like it used to be, but give it enough time and improvements add up.
it's also worth keeping in mind that alot of the 'improvements' are actually advancements in harnesses and tools.
I agree. But notice that you assume that there is a metric with which you can messure improvement. Which is fine if you are measuring against your personal taste.
But it might be that the optimization target itself has a ceiling. If you're training toward human approval ratings from a broad population, you converge toward what median preference selects for. The plateau is baked into what you're measuring against.
It doesn't even need to 'improve' at the same rate to have extraordinary impact in society. Even if the frontier models stayed roughly the same in cost and capability for just 1-2 years, the harnesses and processes built around them would mature. We have not yet metabolized these models. Frankly, a lot of this feels like late 80s early 90s complaints about how office computerization wasn't happening yet--it was, just not at the rate promised by the companies selling computers to businesses. We don't look back at those people in the 80s saying that paper was here to stay as visionaries just because they noticed that propaganda temporarily outran the business environment.
I just wish people would take a step back and think about the timescales here. Language Models are Unsupervised Multitask Learners was in 2019. Here we are seven years later and LOOK AROUND. The landscape is unrecognizable. It's worth thinking about who, in those seven years, had an accurate estimate of the future and whose estimate fundamentally failed. And just as it is valuable to note where propaganda about progress speeds past where we are, we should remember that it is costless to announce that at some unspecified future time all of this will settle down and things will go back to the way they were.
The issue is that before GPT models basically were useless for any conversation. We are literally in science fiction realm. From a text conversation perspective the gap between where we are at and what’s left to get to is relatively small.
In my opinion, the main thing we need to do is have training happen continuously. And probably more real world data (from sensors).