>If performance is the main difference, whatever that means, then basically Go should be reserved for when Rust and other lower level langs cannot be used due to some other constraint?

Or when performance is the main but not the only difference, and there are many other benefits.

>Say I am building some app that I know will be CPU-bound, why choose Go over say... Swift?

Because unless you're building for macOS/iOS, Swift is really a no-go, with lackluster support for other platforms. Plus slow to build and convoluted.