How to structure experiments/scaling and hyperparameter tuning regimes are most of the secret sauce (besides massive compute). If you don't create an experimental ladder to verify scaling and optimize your hyperparameters well, you'll waste a ton of money.

The data is mostly coming from places like Scale/Mercor/etc and net dumps with some filtering and batch prioritization, and RL on verifiable domains like code/math/games.