Screwing around is not a crime, but he also isn't under threat of prison time. He's under threat of losing his job, because he has shown that his judgement cannot be trusted (he had two illicit affairs, one of which was with a subordinate). It's perfectly reasonable to fire someone from a leadership position for having bad judgement.