What you can do in C# today is convert any unsafe pointer to Span whenever you get your hands on it, and pass around slices. You can still drop down to ‘fixed’ when it turns out you need it for performance.
With fixed you do pinning on GC memory, which can have a negative performance effect. You can also do unchecked pointer arithmetics on references with the Unsafe class, which avoids that. A lot of the methods of Span use that internally.
I could say that the perf difference between spans and unsafe pointers in most cases is just zero, if not in favor of spans at times due to the optimized helper methods or just better code generation. Add the safety benefits to the mix, and it's a no brainer. So, unsafe pointers may not have an edge for performance anymore. They might have other uses cases of course: interop, etc.