I were 17, I'd learn how to build LLMs from scratch

This essentially comes down to choosing the search for substance (here in the form of technical depth) over short term gratification and quick wins in life.

I think this kind of mindset should be taught way more in school so that people really appreciate learning a subject deeply.

If I were 77 I'd do the same

My kids CS teacher asked me what they should do after AP CS.

Previously they had a class where they'd build apps for other teachers. Like tracking when clubs are, etc. But now that's become easy for teachers to vibe code themselves.

I suggested they shouldn't prereq this class on CS. Heck invite anyone in interested in "building things" and they can get practice at building apps for other people / themselves. Maybe that becomes a gateway TO CS - people who want to learn how things work under the hood.

I suggested post CS class for the CS people should probably be building an LLM or something :)

i was trying to build llms from scratch at 17. failed miserably because i did not know linear algebra. ended up in a different but adjacent field. when chatgtp got big suddenly there were so many people doing llms and i didnt want to compete like that. so much of what was happening was hype, and that really turned me off. am back to building llms from scratch, but like, its a journey teaching myself all the theory on top of my job. i have decent fundamentals, but i need a better grasp of all the advancements in the field in the past 5 years before i would feel comfortable designing anything. baby steps, essentially. am working on better understanding all the layers of an ai while implementing a rag on my local model as an experiment. 17 year olds should learn whatever theyre interested in but need fundamentals in order to do anything advanced.

A 17yo can train a small GPT this weekend. nanoGPT is a few hundred lines. Understanding why it works is the part that takes a decade.

What would you do if you’re 30 years old now?

As a platform engineer being based mainly out of Australia/Hong Kong, opportunities seem to be getting less unless targeting high frequency trading or banking.

It seems like building a startup with the help of some AI tools might be the best bet.

I'm usually a fan of pg, but this post is ignorant of modern AI technologies. Building an LLM from scratch is both a trivial and a useless exercise. There's probably in the range of 5000 github repos doing exactly that. What makes LLMs work is scale, and what makes engineering and training LLMs hard is also scale. And scale is not something you can achieve in your garage.

If the goal is to understand LLMs deeply, one would be better served by either joining one of the big AI companies or doing a PhD. And to be honest, I think this journey should have been started 5 years ago, because right now there's too much competition.

I wonder at the “worlds” Mr Graham envisions, and what is their cardinality. Is this the only 17 yro reimagining, or is there an army of 17yro, of which this LLM curious persona is but one?

Why is it important to train LLMs or even fine-tune them? LLMs have proven their point, costs are crashing, and there are more of them than most companies need.

The real value is to unlock meaningful insights and directions from existing data that is there inside companies.

I live far outside any tech city, so maybe I do not understand. But working with LLMs full-time, building for clients and tons of own experiments, I see no value in building on LLMs.

So I am 15. Is it worth trying to build my own archive (s-1.site) of strategy ideas? Do I have any real differentiation? Or am I just wasting time? I figure that I can use it as proof that I have some know-how?

I would (and am) going into MLOps. Not just the general infrastructure/systems administration but how to do inference optimization, caching, quantization, memory pinning, vfio passthrough of gpus etc.

Ah nostalgia. At 17, I learned how to write programs in BASIC on a mainframe.

Easier said than done - where do you get the B300s from?

Better to start working with harnesses, evals, statistical analysis, etc. - where you don't need the huge hardware for pre-training etc.

This might be a helpful resource: https://laurentiugabriel.github.io/token-town/

If I was 17 again I would sack off work and study and focus on chasing the opposite sex, without the angst I had at the time. I don't know a single person who regrets having had too much sex when they were young. I would not build an llm, too hard, too expensive to run. Learn how easy life is if your morals allow you to grift money of vcs into your own funds and retire.

When I was 17, I wrote poetry and learned how to play guitar. I’m really happy that I did. That’s what I would do now, were I 17. I feel slightly bad for people who didn’t.

The core point here is that AI is a massive thing (at the moment) so it's probably a good idea to understand it deeply. Not sure why people are so worked up about it.

If you're 17, and seriously curious about how modern day AI works, you might as well just sit down and look at a couple of courses on linear algebra + calculus, machine learning, deep learning, and more LLM specific deep learning. Those courses will teach you how to go from writing your first perceptron to a MVP language model. But also so much more.

As an aside, for someone interested and who's an absolute beginner, can someone please recommend good resources on how to build LLMs from scratch? Thank you in advance.

... and it would be totally pointless.

I mean first that is already what plenty of 17yo are actually doing, because that is what they do at school or in parascholar activities. There are already countless of such tutorials where you can do that in an afternoon.

The pointless part though is precisely why Amazon and others are hunting for rare books, all the low hanging fruits have been picked already so just training a bigger model will simply mean burning more energy and money. Sure training a small one for the basic principle is a great pedagogical thing, training another one, medium, then maybe a large one, is also good in term of learning the process and architecture, but one should not expect it to be useful out of that context.

Pure players are precisely doing everything they can to corner the market by making their own scale unreachable by others. Smaller players with access to lesser infrastructure are thus betting on different market, e.g. embedded systems.

17yos should definitely build their (L)LMs from scratch and whatever bigger model they can train for free, or for cheap, but they should not expect that to bring them any riches.

Seek advice from nice people that you actually know instead of rich people on the internet.

I assume in this hypothetical Cornell just gives away degrees to passersby.

LLMs are the new compilers.

I don't think you can really call yourself a developer unless you at least have an idea how to build a more complex software project like a compiler, and maybe have built a toy one either at uni or for fun.

It's not clear how long this LLM age of AI will last (to be replaced by something better), but nowadays any developer should at least understand the basics of ANNs, and more than just the "hello world" of a cat vs dog CNN. An LLM/Transformer is maybe the equivalent of a compiler in that regard - something that we all use and is complex enough to present a bit of a challenge. You should at least understand the basics of how an LLM is built, and maybe building a toy LLM will/should become the new Comp. Sci. degree toy compiler replacement.

If you’re 17, you might learn hands on knowledge, like tacit knowledge in areas such as lathes, precision engineering, metrology, and other very niche fields. You can also try climbing and explore arts like music, painting, and drawing. And, of course, spend some time in nature.

LLMs are incredibly boring to me as a technology..Not in terms of what it can do, but how it works.

It’s probably wise to learn how to build one to understand what you’re dealing with. However, if I were 17 I would lean how to apply an LLM to a problem instead of strictly building one.

Does anyone have a roadmap ?

An LLM isn’t hard to make - the training data is hard to get and prepare.

The big companies stole the data. The average person can’t do that

certainly better than wasting time with harness and agent workflows that will become irrelevant at the next evolution, same thing happened with 'prompt engineering'

Can any1 share any resource to learn that skill. I want something that has been tried by you. I can too search on the internet...

Paul G is not writing this for a general audience of your run of the mill “engineer” hoping to be employed by someone. He is writing it for future founders. What knowledge / skills you need to develop today to be well positioned to have a startup worthy insight when you are 24.

i am a small fan of pg, nevertheless i find this to be an exceptionally good take and it is strange to me to see so much piling on to this one in particular here.

learning about llms is not useful so that you can make llms later, you want to learn about it so that you can work on next generation architectures. llms before long i imagine will be left in the dust by ebm / physics oriented models especially that can have an embodied understanding of the world. but a lot of things you learn about them are transferable by doing something like this

Why are people so negative about this? It feels like a fun project and at 17 the stakes are not really high. Something one could easily do on summer break in a couple of weeks.

As if you couldn't learn what LLMs are at any Age.

The core technoology is pretty basic, developing a rudimentary understanding for why the individual parts work as well as they do is tricky.

Didn't his swiss watch essay say he'd essentially leave the industry because there will only be bloat from now on?

I read a bit about how LLM works, but as a hobbyist it is pretty frustrating that I won’t be building anything useful without throwing a lot of money at it

I think pg answered the question as “what I’d do as a project” and not “what I’d do as a career.” So the critical comments are kind of missing the point, IMO.

I don’t see why learning how LLMs work is a bad project for a 17 year old.

Optimizing your entire career and the next decade+ of your life on LLMs? Yeah, probably not ideal. It’s almost always a bad idea to make long term decisions based on current trendy things.

And since everyone is using this topic to give their ideal advice to 17 year olds, my advice as a mid-30s guy: seriously consider becoming highly skilled at a specific thing, and don’t be scared off by the idea that it’ll take 5-10-15 years to get there.

When you’re 17-25, the timescale of a decade seems infinite. But it’s really not, and a decade spent “exploring and keeping your options open” sometimes just ends up with you being pretty decent but not amazing at a lot of random things.

Sometimes I wish I had just become a carpenter, chef, electrician, etc. – a specific skill set that leads to mastery over time, rather than the endless exciting-new-thing hamster wheel of working in tech.

I'd just build an agent, it's easier than you think and you'd learn a lot about the "magic" of LLMs.

It’s really disheartening to see how many people don’t know shit about LLMs, by reading the comments… ironic given what OP is trying to say

2 years before everyone was doing custom training. What happened to all those today when frontier models itself become more powerful than custom trained ones?

If only we could train LLMs on commodity hardware, right?

LLMs will take away all jobs except yours, LLM maker. Keep at it, you're safe!

What are the best resources to learn how to build LLMs from scratch for 17 year olds?

I have my opinion on this but I'd like to hear the HN opinion, I will just say one thing:

If you are starting with little knowledge, like a 17 year old would, letting an LLM explain it to you is a terrible idea.

Would building an LLM from scratch imply writing code by hand?

Why do intelligent people still use X? Thanks for the xcancel.com link!

If I were 17, I’d learn how to invest and build financial literacy, and plot potential growth of my networth throughout my life, before even thinking about a career. Then smoke a bowl.

I would not waste my time with yesterday's fad. The next unicorn generator will be something else.