Remainers who insist Brexit voters are racist create a self-fulfilling immigration prophecy

Shortly after the EU referendum, several thousand young people marched through London demanding a rerun. I happened to be sitting next to three of them on a train as I travelled into the capital that morning. They evidently recognised me right away as an Evil Tory Leaver, but we were past Clapham Junction before one of them plucked up the courage to talk to me.

"Are you Daniel Hannan? I just wanted to say that what you've done is terrible. We're not a racist country. You've taken away our future."

"Is that so? Out of interest, can you tell me who the President of the European Commission is?"

"No. What's that got to do with it?"

"Can you name a single European Commissioner, come to that? Do you know what our budget contribution will be this year? Or what the difference is between a Directive and a Regulation?"

She was affronted by the questions. So were her two friends with their "I [heart] EU" placards. They weren't interested in details. For them, it was about values. Are you a decent, internationalist, compassionate person? Or are you a selfish bigot?

Let's leave aside the fact that no one would ever vote on any ballot paper for a "selfish bigot" option. Their determination to approach the issue in terms of character, rather than cost-benefit, explains why they were so upset – and why, even now, some Remain voters struggle to accept the outcome.

In my experience, the 48% who voted Remain fall into two categories. There are those who were making a judgement as to where Britain's best options lay. They could see that the is EU flawed. They were well aware of the corruption, the lack of democracy, the slow growth. But they took the view that, on balance, the disruption of leaving would outweigh the gains. These people, by and large, now want to make a success of things, and are keen to maximise our opportunities.

Then there were those like my companions on South West Trains, for whom the issue was not financial but somehow moral. For them, the EU wasn't the grubby and self-interested body that exists in reality; rather, it was a symbol of something better and purer, an embodiment of the dream of peace among nations. They never heard, because they never wanted to hear, the democratic or economic arguments against membership. As far as they were concerned, the only possible reason for voting Leave was chauvinism.

You can see why they were shocked on the morning of 24 June. Having so defined the question in their own minds, they must have been horrified to think that 52% of their fellow-countrymen were xenophobes. It explains why, since the vote, they have been reluctant to engage in any kind of debate about how to get the best deal. How, after all, can you debate with a bunch of racist numbskulls? Better just to hope for some economic catastrophe that might shock people into reversing their votes at a second referendum.

It's a bizarre strategy. There isn't going to be a second referendum; and if there were, opinion polls point to a stronger Leave vote than last time. It's hardly for me to advise the 48% but, in their position, I'd be focused on getting the best deal. If, for example, it's in our interest to participate in various pan-European research or educational exchange programmes, as several non-EU states do, then make that case. If we benefit from keeping the current EU rules on state aid and non-discrimination in procurement, let's hear it. If certain sectors – financial services, say, or pharma – want to hire qualified professionals from abroad more easily, let's make that one of our objectives.

You can enter into that discussion, or you can demand a second referendum. You can't do both. For example, if you want Britain to remain in programmes like Erasmus and Horizon 2020, fine. But if you want us to keep participating in some EU policies, and contributing to the running costs, you can't simultaneously say "So where's our £350m a week, then, liars?"

The odd thing is that, by insisting that the vote was all about hostility to foreigners, Remainers risk pushing policy in that direction. It is striking that all the nonsense we have heard about limiting numbers of EU doctors and students, let alone obliging companies to list the nationality of their employees, has come from ministers who backed Remain. I wonder whether, like my companions on the train, they're responding to their own caricature of what Leavers are supposed to want.

I still think that the likeliest outcome is a moderate one. Britain will not relate to the EU simply as a benign third country such as , say, South Korea. We're clearly going to repeal the 1972 European Communities Act and become a sovereign state again; but, having done so, we will probably enter into a series of bilateral treaties that replicate some of our existing arrangements. Though we ought to leave the customs union, thus scrapping the EU's Common External Tariff and making possible free trade agreements with faster-growing economies, there are other elements of the Single Market that we might retain – notably the ban on discrimination against another member state's products which is at the heart of it.

There is a small danger, though, that things could go wrong. Over-compensation by ministers who backed Remain, perversely backed by Remainers wanting a bad deal, might lead to a more severe rupture. And while this wouldn't be the end of the world, it seems a pity not to get the best possible deal. What matters more, my Remain friends – making a success of the future, or hoping for the chance to say "I told you so"?

Daniel Hannan has been Conservative MEP for the South East of England since 1999, and is Secretary-General of the Alliance of European Conservatives and Reformists