try disabling at least some debug-info for your dependencies.
use cargo's --timings option to identify the slow crates, then use measureme to figure where rustc is spending time.
try fiddling with detached debug-info, dwp files can be smaller than their constituent dwo files.
ensure really small "template wrappers" are inlined.
lower the number of compilation-units you use.
For incremental builds I have found it can _sometimes_ be effective to split out larger dependencies into a dynlib. You create a dylib crate that re-exports said dependencies and your executable depends on that. Ofc things like generics from the offending libraries will still end up in your executables; Though I think there is a way to mitigate this a little.
This mostly helps with link-time, and saving space if you have a bunch of executables (using the argv[0] trick will help more, though).