You also don't want adversaries to be able to disrupt long-lived streams with bad password guesses, since I think part of Wormhole's security model is it will terminate the session if the other side gets it wrong.