The kernel already dynamically loads kernel modules based on hardware it's running on so most of the junk isn't loaded. But for the stuff you can't dynamically load, it's a tradeoff between stripping the kernel down and supporting the most hardware.
Maybe distros could provide two builds, one for modern hardware only and one for maximum compatibility.
That wouldn't work, because one day you dist-upgrade and your machine doesn't boot
Back in the day, I used to build windows XP as stripped down as possible. Maybe something like that is possible? Distro installer determines what hardware you're running, then only includes what is absolutely necessary?
More annoying the first time you plug in that USB DAC or whatnot, but would reduce the attack surface heavily.
Who am I kidding... someone is going to comment that Dolphin Linux has supported this since 2013.
Back in the mid-2000s I regularly simply built a kernel to match what I was running daily, and kept a stock generic kernel.
I started it in freebsd and later on Linux.
I'm not quite sure why I stopped the practice if I'm honest probably because generic felt good enough at some point and I forgot. Probably because my peripheral use became more chaotic.
That and most of the benefit is RAM footprint which stopped mattering.