Historically speaking marriage did not mean a man agreed to be sexually exclusive with a particular woman.
It meant a woman agreed to be sexually exclusive with a particular man + various legal entitlements. The particular legal entitlements were what distinguished it from concubinage (which was also a woman agreeing to be sexually exclusive with a particular man).
Now that is not what it means in a modern context, but also in a modern context nearly every developed country has below replacement birth rates, meaning that these countries are dying so any particular claims of moral superiority or fitness of the marital/family structure norms of modern societies can basically be entirely disregarded.
So your excuse for cheating in marriage is declining birth rates? Absolutely hilarious.
Stop ignoring the part of the argument in which there was a promise made that was later broken. That is why it is immoral. Respond to that part.