It's not much worse than any other form of mining. Rare earth mining is destructive because mining is destructive and you need lots of ore to produce viable amounts of rare earth minerals. The "separation" aspect that GP is talking about is hard because of that: you need lots of input to produce not much output. (There is a chemical aspect as well; because rare earths all occupy one spot on the periodic table, they are different to separate from each other). But other than that, it's not worse than say coal or tin mining.

Also rare earths (other than promethium) are not themselves radioactive, they just tend to co-occur with uranium and thorium deposits. This is akin to your house having a radon problem though: doesn't mean your house is radioactive, just that you built it on a uranium deposit.