How Britain broke its taboo on blaming immigrants and why that makes Brexit harder

Out of all proportion to its real effects, immigration now determines British politics, exercising political gravity inordinate to its size.

It has caused the Conservative government to aim for a hard Brexit, souring relations with Europe and threatening permanent and irreversible damage to trade. It has paralysed a Labour Party torn between its young urban, multi-ethnic and white ageing post-industrial voters.

This wasn't always the case. In 2005, when I was the foreign policy man at the Conservative Research Department, the Tories ran a general election campaign notorious for its anti-immigrant tone. "Are you thinking what we're thinking?" ran the slogan, beneath the text in looped handwriting script "It's not racist to impose limits on immigration".

We were roundly and rightly defeated, a defeat that galvanised the party into picking the self-consciously modernising outside chance David Cameron as its next leader. What's happened? Have the British people become more hostile to immigration?

Not really, opinion of whether immigration is too high, too low or about right is remarkably stable, though people's beliefs about its importance has followed immigration volumes. But a taboo has been broken.

The urge to divert blame to outsiders is among the most powerful social forces known, and more powerful still in democracies, when foreigners don't have the vote. It's always destructive. At best it diverts us from looking for the real cause of our problems, its worst effects should be too well known to bear repeating. Political systems impose norms against invoking it for good reasons.

That poster from the 2005 election was a failed attempt to break that norm. It failed, as post-election focus groups showed, because we weren't trusted to make the assertion that immigration controls aren't racist. Just as only a Tory home secretary, as it happens the current prime minister, can get away with cutting the police budget, only the Left, guardians of anti-racism, can break the immigration taboo.

The argument begins with pedantic equivocation; it's not racist to object to Eastern European immigration because Poles are white. That reminds me of the Islamist extremist from Manchester University who explained that she couldn't possibly be anti-Semitic because she, an Arab, was herself a Semite.

It continues with a series of falsehoods about Eastern European immigrants: that they are taking British jobs (they are not), depressing wages (on average they don't), putting pressure on the NHS (utter nonsense, because they are young and able-bodied), taking resources from the native population (rubbish - from 2004 to 2014 they paid net £8bn to the exchequer; the native population paid net minus £671bn), are out of work scrounging on benefits (false), are taking in-work benefits at a disproportionate rate (also false, and inconsistent with the idea that they are stealing jobs and top-up benefits for working people).

This may not be racism because the immigrants in question were white, but if they had a different skin colour we'd have no difficulty giving this prejudice the name it deserves.

Facts, as we should all know by now, are not the issue. People's beliefs about immigration express their fear that their situation is to be undermined, or that they are being taken advantage of by members loyal to another tribe. The taboo served to prevent politically influential leaders from amplifying and giving legitimacy to those destructive instincts.

Gordon Brown broke the taboo when he said he wanted "British jobs for British workers" in a conference speech in 2007. Now it hasn't so much been broken as inverted. The new victims of the system aren't ethnic minorities living in major cities, but the white working class living outside them.

There's a lot of evidence that they indeed suffer disproportionately. They attain inferior educational outcomes, less social mobility and often lower pay. They're isolated from many of the benefits globalisation provided the educated and internationally minded. A lot of them indeed feel neglected by the system that they feel has imposed a diverse and open society upon them. And they are particularly worried about immigration.

This has provided a reason, authorised by a Left unable to convey to them how their lives might be improved, to retreat into caring for their prejudices. Now the received view has transformed: it isn't pandering to racism to suggest policy conforms to their views, but rightful concern for the left behind.

Social mobility has replaced racial superiority as the motivating cause, so people aren't stopped by the taboo against racism from expressing anti-immigrant feelings. Anyone who points out the truth - that these are attitudes based on fear of and generalisations about other groups of people - opens themselves up to attacks that they are privileged and out of touch.

There's nothing as dangerous in Britain's oppositional system as a political consensus. Bad enough when, as with the Iraq War or the financial crisis, it was the consensus of informed experts. Much worse when it derives from ignorance and prejudice about the very foreigners Britain has to persuade to offer good Brexit terms.

Garvan Walshe is a former national and international security policy adviser to the Conservative Party, and a columnist for Conservative Home. He tweets at: @garvanwalshe.