I'm not too sure about that. Personally I feel like for myself it's tied up with self esteem. I have confidence in my skills, just never when there's strangers watching. Also to maintain the world state in your head and avoid stepping on toes etc.
I can feel my heart rate go up. Sometimes when asking or answering questions in the lecture hall I start quivering. Just the thought of standing up, preparing my question and follow ups can cause this.
You may be right in that it's an excuse. But at the same time there's this disgusting physical reaction and yes it's easier to give in than to stand up.
It's easier to interact with loved ones and close friends because you know they know you and won't judge if you say something wrong etc. But not so for any other interaction.