alt Hacker NewsThe reason why I don't see the promise for ML-only applications is that the coordination backprop requires comes very cheap to us.
"Much easier given the right device" - the "right" there just isn't shaped like the devices we actually build. And the price of "not having backprop" is usually expending more FLOPs, getting worse sample efficiency, etc.
The biggest "device" that doesn't do backprop is the brain, and that's because the brain doesn't have the connectivity or the coordination to pull it off. Both of those are "expensive" for something like it to implement. Cheap for us though. We aren't stuck with neurons that only get locally available information and have to implement learning rules based on that. So, skill issue?
Replies
I mean co-ordination requires energy though. The brain wattage looks at GPUs and says "skill issue". But you're right that we look at natural energy production techniques and say "skill issue"
What about applications like distributed computing using volunteer computers, instead of datacenters? You can't really do that with normal backprop approaches. E.g. for LC0 the training data is generated in a distributed manner, but the neural nets themselves are trained centrally.