rust would be pretty unusable without references. affine lambda calculus isn’t even turing complete. however, you’re right that a borrow checker is unnecessary, as uniqueness types (the technical term for types that guarantee single ownership) are implemented in clean and idris without a borrow checker. the borrow checker mainly exists because it dramatically increases the number of valid programs.
Supporting single-ownership in a language doesn't mean you can't have opt-in copyability and/or multiple-ownership. This is how Rust already works, and is independent of the borrow checker.
If we consider a Rust-like language without the borrow checker, it's obviously still Turing-complete. For functions that take references as parameters, instead you would simply pass ownership of the value back to the caller as part of the return value. And for structs that hold references, you would instead have them hold reference-counted handles. The former case is merely less convenient, and the latter case is merely less efficient.