As a man who's never wanted kids, and is now getting to an age where it probably wouldn't be a good idea, those weren't really the big factors for me.

Having a kid is just an unfathomably large commitment. If you bring a kid into the world, I believe you're responsible for creating the conditions where that kid can grow into a healthy, well adjusted adult, and that's seemed like an increasingly impossible commitment for the past few decades.