> why isn't this done

It IS done with every new model release. Why do you think SpaceXAI bought Cursor? And then immediately had a huge jump in capability with their next model version?

It's just done in large batches for economic reasons.

> backprop fundamentally wants the entire data set in every pass

I'm genuinely unsure what you mean, it's not even possible to run backprop in this way?