It is neither racist of xenophobic, and you using that as a argument is an extreme strawman.
There is certainly a class that has been disrupted over the last 20-30 years to the point where their lives gone worse.
It would likely have been protected with trade barriers and immigration barriers.
Om the other hand, I don't believe protectionism has ever been the solution. Why should the domestic labor be protected over international labour?
Increasing inequality is, however, one of the greatest threats so society.
Trade barriers are a different thing than immigration though. I believe there is much more evidence that trade displaced traditional jobs than that immigration led to an oversupply of labour.