Because it will be very messy and involve a lot of suffering if unmanaged.

The population just doesn't disappear, it can pretty quickly shrink in just a generation or two leaving huge amounts of infrastructure unmaintained and falling apart with huge amounts of debt that will ensure what remains of society ends up in chaos.

That and the most likely part of the population to shrink is the ones we consider more stable and rational. Cults and religious breeding groups will increasing become the majority of the population leading to some 'interesting fun times'.

The impact of humans on most other species, and the environment in general, isn't exactly positive.

That's the neat part! You don't have to care. It will either way. But it's not looking great for liberalism (and that's good thing).