These aren’t raw base models they are the result of a ton of RLHF and various adjustments.

Bitter lesson wildly overstated in this context.

While I think this is true, remember as we get more efficient we just decide to scale even bigger. So more GPUs, and more efficient.

I agree with the sibling comment, effiency is probably the more important component at this point. We are hitting not just a practical engineering roadblock for scaling with current technology, I think we have definitely hit a financial and logistical roadblock for up scaling with the number of GPUs (on an immediate basis)

There goes my plan to buy a PC for the next decade

Not always, in some cases, changing to a higher reasoning makes the AI doubt itself too much, and skip over the correct answer by overcomplicating the problem and polluting the context.

It would be nice to see on which categories of problems the extra thinking makes it better and on which it makes it worse.

Kind of refreshing though that the "throw more processing at it" scaling we saw in the 90s has returned in a different way. For a while we were really bottlenecked in our advances by relatively low levels of parallelism (most software used by your average user doesn't scale cleanly with more than a few threads).

I mean, theoretically you can solve every finitary problem with a brute force solution...

Richard Sutton specifically states that the search has to be smart. We know that the brain uses recurrent connections and is shallow. I think a lot more money has to go into architecture. Feed Forward transformers can only scale so far

We’re definitely going to need a lot of Gpu’s

Or a new model. The human brain does far more with far less.

Or a lot better efficiency.

I said a few months ago, "man, Opus is great, but sometimes when talking with it I have the feeling like, this thing should be about 10 times bigger."

When Mythos was announced after that, I was pleasantly surprised to hear about it. But when it turned out to be only two times bigger, I was a little disappointed!

(I am even more disappointed with the safety filters, but that's kind of a separate discussion... "Fortunately" I find that I can usually edit my prompt by single character and get through...)

> Dramatic difference

Isn't this just the difference between getting 0 right and getting 1 right?

And a lot more electricity to power them.

And dozens of data centers in every state so tokens are dirt cheap.

> We are probably going to need a lot more GPUs.

Or a breakthrough in algorithms etc.

The human brain, heck all bio brains, are proof that you don't need a lot of power or size for intelligence.

This isn’t really how it works anymore. Agents rely heavily on tool use and the agentic harness to perform tasks. Pre-training is no longer very effective.