It's not just syscalls. If you trust all of the running code, you can use a single address space for all applications and even the kernel, which lets you avoid TLB flushes.

Whether that's still a good idea given Spectre is another question.