It's not the affair, exactly. It's that he violated HR policy (which incidentally was by way of having that affair).