I think a little reasoning at global scale easily arrives at a defence of the current traffic law depending on risk tolerance, but perhaps an analogy will help: most societies believe (or at least enforce in a manner that reflects belief) that walking around with a handgun out in your hand should be illegal. In truth no harm is done until one pulls the trigger, and there is certainly benefit: it is much faster to stop a criminal when you're ready at hand.

The reasons we don't do that are manifold, but at least a few are analogous:

* legibility: we don't need just lack of harm, we require common knowledge that harm is unlikely in order for society to work with frictionlessness we desire

* distinguishability: at some percentage of accidental behaviour, we must constrain all people because we don't have a mechanism to determine who will likely cause it and who won't

* reversibility: for sufficient harm, it is better to restrict the error condition than it is to punish

Because we know we cannot bring the dead back to life, and no amount of prison will bring solace to their loved ones, we have decided that doing things that are high-risk to others is not permissible. Given this framework for the moral concern, it's just an optimization problem. The question then becomes what fraction of pedestrians killed in crosswalks is acceptable, or even what fraction of pedestrians following the law killed in crosswalks is acceptable. Some societies believe this should be zero (hence the amusingly named Vision Zero and so on as practised in Northern Europe). Others believe this should be fairly high (like the US) because the utility loss from constraint is too high.

Now the handgun case has a very high number for potential risk, so it's obvious why most societies have that law. The crossing point of risk for almost everyone is below it, consequently most agree. The question then becomes what your crossing point for risk is and whether the number of accidental deaths is above your threshold or below your threshold. But in either case, I don't think the argument "until they hurt someone, no harm is done, and therefore it should be permissible" holds, for if it did, surely we would allow for people walking around with handguns, perhaps even pointed directly in front of them as they walk, so long as they do not pull the trigger. And that seems to be an absurdity.