I had the same inspiration!

I spent a days reviewing the lecture notes for CS336: Language Modeling from Scratch - and then trained a nanoGPT-esque model in PyTorch.

I'd recommend trying it for those who are curious. Computational bottlenecks become much more intuitive when you've looked at the overall process.