alt Hacker NewsIt's still important to support research in this direction cause us meatbags cost a LOT less energy to train than GPTs even if you assume that it takes 30 years to train a PhD. Our current training methods honestly leave a lot to be desired. We almost certainly dont do full backpropagation.
Replies
I'm sure we don't literally do backpropagation, but differential equations that settle toward stationary states show up a lot in biology
You need to factor in the cost of training all those PhDs that never end up producing much of interest though you don't get to pick the best afterwards and claim all it took was to train him/her. Plus, once trained, it's productive only for 6 subjective hours per day (including weekends and holidays). Might have costed gazillions to train a frontier LLM but it works for millions of hours/day.
PHD inference doesn't scale, though.
I think it's fun back-of-the-napkin sometimes to compare meat to matmuls but ultimately fallacious to its core, making it an intellectual tarpit.
It's much harder to argue with the math and empirical results.
I’m not sure that’s true. We have literal billions of years of training.