alt Hacker NewsYou basically need both. Parameters and good post training. If you keep on growing both, you’ll have good models.
LLMs are still surprisingly “easy”. You need maybe a couple dozens of right people, a lot of good quality data and a lot of GPU that you know how to operate. There’s relatively little “secret sauce” needed.
Replies
FergusArgyll • today at 6:31 AM
I think there's still a ton of secret sauce needed for serving them economically
➕ show 1 reply
How to structure experiments/scaling and hyperparameter tuning regimes are most of the secret sauce (besides massive compute). If you don't create an experimental ladder to verify scaling and optimize your hyperparameters well, you'll waste a ton of money.
The data is mostly coming from places like Scale/Mercor/etc and net dumps with some filtering and batch prioritization, and RL on verifiable domains like code/math/games.