The borrowchecker is not the source of the bulk of the language's slowness. It takes a fairly small fraction of the total time, I think I remember the number was 5%-15% depending on the project.

The majority of time is spent between the trait solver and LLVM optimization passes. There is work being done to improve the trait solver and to produce more efficient LLIR code from the get-go (so LLVM has less to optimize).