I think it actually must be because they aren't from the western world.
THere are far more illegal overstayers who are here than boat people, but most of them are from England, Ireland and the USA, they don't pay any tax, cash in hand etc. But no one cares, no one ever jumps up and down regarding them screaming "they are stealing our jobs".
What other conclusion could possibly be drawn from that than it is more about race than visas??