There's a difference in intention between ex-pat and immigrant. Ex-pat's tend to think of themselves as being wherever they are temporarily, but intending to return to their home country. Immigrants desire is to make wherever they are their new home country.

If you're saying that people who have permanently left the US call themselves ex-pats, that is news to me, and I can understand the confusion.