alt Hacker News... and it would be totally pointless.
I mean first that is already what plenty of 17yo are actually doing, because that is what they do at school or in parascholar activities. There are already countless of such tutorials where you can do that in an afternoon.
The pointless part though is precisely why Amazon and others are hunting for rare books, all the low hanging fruits have been picked already so just training a bigger model will simply mean burning more energy and money. Sure training a small one for the basic principle is a great pedagogical thing, training another one, medium, then maybe a large one, is also good in term of learning the process and architecture, but one should not expect it to be useful out of that context.
Pure players are precisely doing everything they can to corner the market by making their own scale unreachable by others. Smaller players with access to lesser infrastructure are thus betting on different market, e.g. embedded systems.
17yos should definitely build their (L)LMs from scratch and whatever bigger model they can train for free, or for cheap, but they should not expect that to bring them any riches.
Replies
It seems obvious to me that no matter what your age right now, you should want to be as broadly educated and intellectually curious as possible.
You should want to train a LLM from scratch as an intellectual curiosity itch that needs to be scratched.
The idea of learning one hot skill that has a pot of gold waiting at the end of it was a brief moment in time that came and went.
When I was 17, we would have said obviously support vector machines are the future. Neural networks overfit and don't work.
Why would 17 year old do something that only brings them money? I do not think Mr Graham here is advocating for the path that makes most money as a result of learning how to train a model. I assume that tinkering and learning about LLMs is what enterprising 17 year olds will do to discover ways they can get a competitive edge or further the SotA with their insights further down the line.