That's the main downside of divorce, not of having children or the custody split.