I agree it shouldn't really matter if there's no C++ features in play, but I suppose third party headers could bite you if they use #ifdef __cplusplus to guard optional C++ extensions on top of their basic C interface. In that case the compiler could be dealing with dramatically more complex code when you build in C++ mode.
Maybe it is similar for the same compiler (but one should check, I suspect C could still be faster), but then there are much more C compilers. For example, TCC is a lot faster than GCC.