Worth noting for anyone reaching for this in practice rather than out of curiosity: several standard library implementations (Python's math.factorial is one) already use a divide-and-conquer multiplication scheme instead of naive sequential multiplication for exactly this reason, so you often get most of that speedup for free without implementing prime swing yourself.
(I just undead'ed this comment; can't see why it was downvoted.)