The issue with billionaires is that some of them got insanely rich while being net negative from a societal perspective.
The most famous ones ended-up in prison (Sam Bankman Fried, Elizabeth Holmes, Jeffrey Epstein, Bernie Madoff) but anyone with a basic grasp of statistics and criminal behavior know that many others will escape the justice system forever.
It does not mean that all billionaires are bad, the criminals are not the majority, but there are enough criminals to justify skepticism and scrutiny.