The existence of even one irrational actor turns it into a prisoner's dilemma. The payoff matrix in a prisoner's dilemma is defined by the value expected by each specific player. If a single player falsely evaluates the expected value of building ASI as positive, every other player is forced to race for ASI even if they correctly evaluate it as negative.

Business as usual beats probable extinction, but probable extinction with a small chance of becoming a living god beats probable extinction with a small chance of becoming a slave.