I found that reducing my "Linux" lines from ~21000 (including net-pf-16-proto-21) down to those ~3000 I might actually use (e.g. udp_tunnel) to be a fairly effective method of not having to care about each and every newly discovered memory safety hazard.
I remember my earlier days of Linux of having to compile a kernel module to read from cdrom. Seems like Linux has gone too far in the other direction of having modules that you will probably never need.