Natural reproduction already has a lot of randomness and failure built into it. It's not like egg/sperm selection is a perfect quality-control system. IVF has also been around for decades, and while it definitely needs monitoring, we haven't seen some obvious collapse in health from it