alt Hacker NewsI think this is decisively the wrong way to think about it. Yes, layering hypotheticals like that means that any one scenario is extremely unlikely to be the thing that gets you, but that doesn't mean the shape of the problem is wrong.
It's like arguing with someone who doesn't believe in using seat belts when driving. "Why should I put them on?" they say, and when you try to explain what might go wrong they won't listen to any explanation that isn't a hyper-concrete hypothetical. So finally you give in and say, "Well, when we get onto the highway, a truck might lose control and hit us", and their response is "I don't think that's very likely, it seems highly improbable that today we will be hit by a truck when getting on the highway".
I agree with OP that this seems like the kind of thing where the unknown unknowns are so great that the correct approach is serious caution, and that any demand to know exactly how or why it will go wrong, falls in the trap where every specific example is very unlikely to be the thing that goes wrong, but still in total there's like an 80% chance that it goes horribly wrong. I don't know if we have the terminology to talk about this kind of failure mode. "You shouldn't play God" maybe? At least you shouldn't ask for specific examples of how things could go wrong, if you're going to turn around and claim each one highly improbable.
Replies
"To play God" isn't just to do something spectacular which wasn't possible before. It's to use other people - to put yourself on a higher teleological level than other people, declaring their purpose (whatever it is) as fully subordinate to your purpose (whatever it is).
As if they were a tool you created. Obviously, if I carve a piece of wood into a spoon, it's I who get to say that what used to be a piece of wood now has a purpose of moving soup to mouths. The carved piece of wood now has a purpose, but it's wholly subordinate to my purpose (whatever it is).
You don't have to actually design people to play God - you can subordinate others to your purpose without doing that, that's what various God-kings in history did. But it certainly gives you a head start if you make them.
You can make something and don't subordinate it to your purpose. In our culture, we see children that way. We claim, basically, that we didn't deliberately design a child, we only obeyed our own natures without really having much choice in the matter, and thus we and our child is on the same teleological level.
This is not a cultural universal. In many cultures, parents (in particular fathers) would say basically "I made you, so you must do as I please, you have no reason for existing except for my purposes". It was a hard won battle for our culture to assert that children matter for their own sake and not just for their parent's.
Many things in this thread makes me want to say "Y'all MFs need Jesus". I won't say that, but I will say that you should stop and think about why it is that the Catholic church is so difficult about contraception, why Christians in general have historically made such a big deal of the difference "born of God" vs "Created by God" (arianism, etc.), and what that story of Abraham and Isaac was really about. Whether you agree or not, there's much about other people you will never understand if you don't think about teleological levels.
The history of humanity, I say of hominids even, was defined by humans playing it unsafe - migrating, sailing, inventing bombs, you name it. We played god before even we invented gods, and reached this point in time. Should we say "this is best we can do, let's stop everything"? Nah, not likely.
"I don't know if we have the terminology to talk about this kind of failure mode."
We actually have and is called RISK.
RISK = Probability * Damage.
Applied to the seatbelt event we have a death level damage and a high probability of happening given recent studies, so using a simple belt could easily save you from deadly accidents.
Applied to any unrealistic scenario we have insane level damages but also an incredibly low probability (near 0) so RISK = ~0%