There is a algorithm call Prime Swing Factorial that can compute large factorials exactly in arbitrary precision math using prime factorization. Like 10000000! in under second depending of how optimized the math library it. Probably like 100x faster than the normal method.
Worth noting for anyone reaching for this in practice rather than out of curiosity: several standard library implementations (Python's math.factorial is one) already use a divide-and-conquer multiplication scheme instead of naive sequential multiplication for exactly this reason, so you often get most of that speedup for free without implementing prime swing yourself.
With Lisp you can use iterative algos and get that under a second too. SBCL can be ridiculously fast; and if you optimize the compilation for integers... the speed gets really close to your solution.