Because in a large number of cases, people are conflating economic situation with culture or race. And many of these so-called cultural or racial components disappear in the "home" country when you look at people in the middle class and above. Just like violent crime rates drop massively for those same races and cultures in the country they have immigrated to when they gain economic independence.

It's quite possible that high levels of immigration of unskilled or otherwise unemployable people is a bad thing for a country that shouldn't be allowed. That is a very different thing than acting like the reason it is a bad idea is because of their culture or race.