There's plenty of beautiful math there, but the relationship to what our neocortex does is pretty distant. Individual biological neurons can do fairly complicated things, including compute 10-bit parity functions (you would normally need a 3-layer MLP with a bunch of digital neurons to do this). And they don't seem to use backpropagation for learning.

Could you provide more detail? My understanding is that the neocortex is predominantly focused on forwards simulation, which seems distinct to how transformers operate.

> and it's the same math our neocortex runs on.

The math is based upon theories of how the brain work, but even if those theories are right, this math is a great simplification and subset of what organic brains do.