This is a consequence of the fact that they are what constitute the legible identity of a given society. The middle and underclasses in any given society just reflect the structures and circumstances created by the elite. The middle classes of every society simply desire psychological security i.e the ability to follow a track laid out before them, the underclasses of every society simply want to fulfill their base desires. The specific character of a society comes from the elite strata who desire self actualization, the character of a society being what self actualization means in that society.

They are precisely the ones that can damage society because their beliefs, activities, desires are what the society actually is.

They tend to be the one that define it. What you consider damage is their (our, frankly) bread and butter.