> isn't a perfect fit for these algorithms but it's relatively close

I don't think that's true. The best fit out of what's presently available perhaps. Inference is almost entirely memory bandwidth bound at present, to the extent that GPUs with HBM have a massive advantage over those with GDDR. TPUs appear to be a much better overall design.

I expect that a hypothetical advance in fabrication enabling processing elements to be placed directly adjacent to dense RAM on the same silicon (not merely in the same package) would be superior in all regards.