I can't help but think the borrow checker alone would slow this down by at least 1 or 2 orders of magnitude.
Your intuition would be wrong: the borrow checker does not take much time at all.
Again, as this been often repeated, and backed up with data, the borrow-checker is a tiny fraction of a Rust apps build time, the biggest chunk of time is spent in LLVM.
The borrow checker is really not that expensive. On a random example, a release build of the regex crate, I see <1% of time spent in borrowck. >80% is spent in codegen and LLVM.