logoalt Hacker News

tibbar • today at 12:00 PM • 2 replies • view on HN

Internet data is only good for some parts of model training. That is why there is now a huge cottage industry that creates custom datasets, RL envs, etc.

Incidentally, Google is often considered to be extremely good at pretraining, which is just the thing internet data is good for.


Replies

andy_ppp • today at 1:29 PM

Yes I'm very much talking about data used in the RL section, I'm sure Google are indeed very good at pre-training.

htrp • today at 3:55 PM

Too bad pretraining hasn't been relevant for model performance since 2025