I agree more with Yann LeCunn's salty reply. Over long run, knowing how autoregressive language models work from scratch will be just one step in having foundational understanding, and they might become dated... like knowing how a CRT monitor work. Something of historical interest and good for learning, but not crucial to being well-rounded.

There are other types of models like diffusion models right now that are showing more efficiency and have a higher ceiling for improvement. Understanding math and fundamentals are more important.