> Somewhat surprisingly the gains do not seems to come from more compute

I think this could form part of an argument that acceleration is slowing. It's only because we are not making as much improvement out of hardware gains that we're having to find the improvements in more complex architecture and more tricky to create training data.

The fact that the gains are not coming from the traditional place gains come from in technology might be a sign that a high percentage of the low hanging fruit has now been picked.