After the news about a potential foreign compromise, disclosure of an affair with a subordinate is pretty damning. One can be discounted as poor judgement. Two is a pattern that's hard to dismiss. I wonder if we'll learn of other shenanigans or whether kicking him out stops future (possible) disclosures.
Poor judgment? He was larger than life persona, many times compared to Musk (minus huge Twitter follower) who is also very well known for his extramarital affairs.