I think calls to violence usually only count as threat when they're targeted at a specific person. To me, hate speech is about creating an unsafe, threatening or oppressive situation for a vulnerable demographic group. Like saying a specific ethic group is a threat to society, or general attacks against women in general. Thing that, when repeated enough, will normalise the idea of violence or oppression against that group.

But what about similar speech against powerful groups, like billionaires? Well, they're not vulnerable or marginalised, they already have a disproportionally large voice, so they're not easily silenced. And of course being a billionaire is a choice; they can abdicate from their wealth in a way people can't do from their gender or ethnic group. So it's not the same. That said, calls for violence against them might still count as hate speech or be considered specific enough to count as threats.

There is obviously some fuzziness, but there also is with other limits to free speech, and there are courts to ultimately adjudicate that.