> Like why can't the CPU compete against a GPU like that for GPU-y tasks - or can it?

Others have taken a stab at the actual differences, but there is a deeper fundamental reason.

A CPU is optimized for low latency of operations. They are designed to complete a given piece of code as fast as possible. There are some affordances for throughput, such as SIMD, but even those are designed to only be as good as they can without compromising the low-latency design of the core.

And the reason this cannot compete with GPUs in throughput loads is that after a point, completing a single task 2x as fast costs a lot more than 2x the transistors and power. CPUs chase that curve as high as practical, GPUs stop once it no longer makes sense for throughput. This is not just clock speed (though it is also clock speed, modern GPUs hang around in the 2.5GHz area while CPUs are about twice that), but especially their ability to hide memory latency, and ILP. CPUs spend big on being able to issue, execute and retire multiple instructions from the same stream, with complex reordering and more than half a dozen execution units per thread, while GPUs are either scalar within a thread, or maybe dual issue. A CPU has a cache hierarchy optimized for bringing average memory latency down, while GPUs just juggle more threads and use them to get something to execute when waiting for memory.