>I doubt that’s the case. My guess is we’ll hit asymptomatic returns from transformers, but price-to-train will fall at moore’s law.

There hasn't been a real Moore's law for a good while even before LLMs.

And memory isn't getting less expensive either...