Incentives can produce locally rational and globally harmful behavior. (See also: "Moloch", although I'm less of a fan of the concept lately since it focuses exclusively on the harmful aspects of emergent systems, deleting the neutral and positive ones in the process.)

A relevant metaphor is the nuclear arms race. Nobody wants a world full of nukes pointed at each other, but if the other guy is gonna do it...

The argument in this case is, we can't slow down, because China certainly won't, and do you want those guys to build the machine God first? So it's "rational" to maximize p(accidental_skynet).