I had previously got around this by using Anaconda but I don't really like the amount of crap that brings in either, and also it leads to a dev environment that doesn't look anything like production so that sucks too... as a result of which I'm back in the boat you're describing.
It's still better than it was 10 years ago (compiling everything from source wasn't uncommon even then!) but it's painful. Comparisons to other interpreted languages aren't really fair though, if JS/TS had as many compiled C libraries, they'd be in exactly the same positions. Go doesn't have the same compiled library thing because of the focus of excluding C dependencies wherever possible.