I was of this conviction as well. However, since I installed CachyOS on my old desktop (6800k+32gib DDR4, SSDs) I have comparison material with my AMD 5900X and the difference is night and day.
Booting to the password prompt for the encrypted drive alone takes 10+s versus 1-2s. Then booting into desktop is also considerably slower (but oddly not 5x).
Then performance compiling is miles apart with 20x at minimum.
Yes, the “native” apps are still snappy but at any time you require compute you will feel the lack of power. Funnily enough it doesn’t seem to matter too much for the games that I play.
Of course there’s a speed difference, but boot times are only occasionally dealt with and only the tiny handful of people capable of developing software will ever need to compile anything.
Also worth noting that if computers had been frozen at Skylake speeds for the past decade, it’s almost a certainty that more attention would’ve been paid to performance since that would’ve been the only way to make computers faster. Countless bits of low hanging fruit in optimization have been ignored because power increases have papered over them, and cumulatively their impact is significant.