In reality there is a continuum of concern and relaxedness, but we can take some moderate point on that scale to define the boundary of the green set.
Now it can be (and is) argued, particularly by people who are relaxed about immigration that by being concerned about it - going in the green circle - you are allying yourself with the bigots - blue circle. This is an effective argument for many people because it pushes buttons in the brain relating to tribal instincts not to ally with the enemy. The consequences of using this sort of argument is that it squeezes the green space in the above diagram, driving people who are swayed either out of the green circle or into the blue one. That is to say, it polarizes opinion. This, then is a very double-edged tactic.
I think it is high time we all did more to depolarize this issue, which in turn would make more room to listen to people's genuine concerns and preferences - whether justified by the facts or based on tabloid hysteria - and less room for the nazis.
Update: I should have mentioned that Mrs Duffy complained that "you can't talk about immigration". This is absurd - everybody is talking about it, including Mrs Duffy. However the sense that you shouldn't talk about it is real, and is a direct consequence of the squeeze on the green circle that I discussed above. The squeeze is intended to shrink the green circle by creating this sort of discomfort. Rather than being snotty about Mrs Duffy's complaint, we should recognise that she is feeling, but quite reasonably resisting, this divisive argument.
We can also show the set of lifelong Labour voters on the diagram.
Mr Brown and Mrs Duffy as far as we know both lie in the brown area - the overlap of the red and green circles. Given therefore their proximity, it is a disgrace for either to damn the other. This will cost Gordon Brown, and rightly so.
There you have it. But let's have one more picture.