Same with leap days though?
The point is that it's weird that we handle a day every 4 years off in a different way to a couple of second being off.
The leap day system handles the mean, the leap seconds handle the variance around the mean. The need for leap seconds is not predictable—they zero out accumulated error.
Leap days are predictable whereas leap seconds are not.
Don't we handle them mostly the same? In a leap year, the month of February gets a 29th day, labeled 29. On a leap second, one of the minutes gets a 61st second, labeled 60. Or we drop the 60th second, and second 58 is followed by second 00 of the next minute.
The notable differences are that
1) the leap second happens at the same time globally (23:59:60 UTC), while leap days start at 00:00 local time
2) leap seconds happen at irregular intervals
3) leap seconds are nearly universally implemented wrong, because the ability to show :60 on a second display for for one second at most twice per year is just not worth the implementation complexity
You could argue about 1, but the alternative would lead to much more complicated timezone math (time zones can be an additional one second apart from each other depending on whether the leap second is already applied) for very limited benefit. Number 2 seems unavoidable, and 3 is entirely unintended, just the way things have worked out in real life