First, his actions were alleged to have impact on national security, but that is not for sure. It is for sure that he was acting inappropriately as a boss, though. Second, I suspect that it is both events which led to his firing. If the affair with the subordinate had come out first, and then the affair with the alleged spy, I suspect he still would've been given grace for the first but not the second event.