You have misunderstood what structural racism is. It is not about the majority of people being racist. Is about the systems being constructed in ways that lead to racist outcomes. You can have a society with zero racist individuals and if they continue to enact the racist systems (perhaps created by racist folks long dead) you'll have structural racism. I don't disagree with the idea that the mis-understanding you have is widespread though, and would certainly be a cause for folks not being comfortable with the idea (as they have mis-understood it).