At risk of over-simplifying, GPUs are wider with limited computational expressiveness and higher memory bandwidth while CPUs are highly expressive computationally (and better connected to I/O) but with lower memory bandwidth. GPUs are less sensitive to memory latency by necessity. Even AVX-512 is highly flexible when inter-mixed with scalar code. GPUs get their very high register width by restricting what the cores are capable of doing efficiently.

Current CPU cores do two AVX-512 operations per cycle. If you can saturate this you’ll often run out of memory bandwidth on CPUs because of lower bandwidth compared to GPUs. In principle, if you bought a 192-core processor you’d have 6,144 GPU-ish cores of 32-bit operations, and they would run at a significantly higher clock rate than a GPU. It would not be competitive with a GPU for the kinds of things GPUs are good at it but it wouldn’t be as far off as you might assume. For some types of code, AVX-512 is unambiguously better.

Horses for courses. GPUs and CPUs were optimized for different things but their capabilities have slowly been converging over time. They all work from the same transistor budgets, the differences are where the tradeoffs are made.

There is a pithy silicon architecture tradeoff trilemma to be made regarding CPUs, GPUs, and barrel processors.