> Success of autoregressive next token prediction feels like a stroke of luck.

Hard disagree. Self-attention + MLP was a godsend because it keeps scaling with compute ("the curve goes up and up and up"). Autoregressive next-token prediction turned out to be the most compute efficient way to train it.

We are almost 10 years into this, and it's still here (and in a field that moves at light speed!).

This is not a stroke of luck. It's here to stay.