The concept that a man can even “cheat” on his wife is a modern idea.
You know what else is a "modern idea"? That you can't sleep with a 12 year old and that you can't own slaves. Both were considered normal for most of human history. So I guess they're ok too, by your logic?
It violates the agreement he had with his wife. Modern expectations are that you do not commit infidelity.