Internet data is only good for some parts of model training. That is why there is now a huge cottage industry that creates custom datasets, RL envs, etc.
Incidentally, Google is often considered to be extremely good at pretraining, which is just the thing internet data is good for.
Too bad pretraining hasn't been relevant for model performance since 2025
Yes I'm very much talking about data used in the RL section, I'm sure Google are indeed very good at pre-training.