The term "safety" (also cf. "violence") has become extremely abused in the modern discourse. From what I can tell, the modal reason to "feel unsafe" nowadays is hearing an opinion the complainant deems unacceptable to hold, or the physical presence of a person the complainant associates with such thought (assuming that said person belongs to a group that is socially acceptable to target).
See also fearing the predictable consequences of their own actions.
This is definitely the case. We've conflated physical safety with emotional and mental safety, and while I think it's okay to use the word safety to describe all three, they're radically different things. Physical safety is paramount, and because of this, people want to use the phrase "safety" as sort of a thought-terminating cliché to get people to change their behavior.