Toby Young was a tribal offering on the altar of sacred values

In politics, it's the little things that count. Three long-serving ministers left the Cabinet yesterday, but today's news was dominated by Toby Young's decision not to take up a part-time position as one of 15 non-executive members of a higher education watchdog.

Toby's self-denial led BBC bulletins, and was the favourite subject on every radio phone-in. A lot of people evidently believe that having posted obnoxious tweets disqualifies you from serving in any state position, however minor, whereas supporting Hezbollah, the Soviet Union and the IRA is no bar to being prime minister.

My point, though, is not that people have double standards: that is part of the human condition. Nor is it to complain about online mobs presuming to vet public appointments. That, alas, is nothing new either. Nor yet is it to point out that Toby Young was qualified for the post: he had set up four schools and worked for years to widen access to universities. Nor, indeed, is it to point out the Left-leaning bias of our quangos, even eight years after the last Labour government. It isn't even to warn outraged youngsters against the precedent they are setting. Do they really want a world where jobs are dependent on your past tweets?

No, I simply want to draw attention to the disproportionate way in which we huff and puff about issues that don't affect anything much. How much difference is made by one of 15 directors of a state agency? How much, by contrast, is made by the minister who ends up running our pensions, negotiating Brexit or overseeing the NHS? But which is getting more column inches?

Henry Kissinger observed that academic disputes were as bitter as they were because so little was at stake. A similar rule applies to politics. Consider, for example, Theresa May's announcement that she will not, after all, allow time for a vote on repealing the hunting ban. Again, the practical impact is minimal. There was no majority in the House of Commons for repeal; and, in any case, the ban has made little difference, as hunts have found ways to remain technically within the letter of the law. But, during last year's election campaign, the PM's reasonable promise to allow MPs a say became, for some voters, shorthand for the monstrous sadism that they always half suspected in Tories.

Several psephologists argue that this was the issue that cost Mrs May her majority. If they are right, then the future of our country – its taxes, its houses, its schools – has been materially affected by people objecting to a vote that would, in all likelihood, have made no difference to anything.

It's the same story over and over again. We seem to care less about our post-EU trading relationships than about the colour of our passports. Each side affects not to be bothered, and mocks the other for supposedly getting so exercised about it. But, measured by airtime and Twitter spats, both are far more bothered than they admit.

Think how much energy was expended over the same-sex marriage legislation. A written question in Parliament asked what legal consequences would follow, since civil partnerships already existed (and had attracted far less controversy). Two things, came the reply: adultery could now be cited as a cause for divorce, and same-sex partners of members of the House of Lords would now be allowed to use courtesy titles.

Of course, for both sides, it wasn't really about how to address the same-sex partner of a peer. It was about, respectively, opposing bigotry or defending traditional values against fanatics. In other words, it was about signalling that you disliked the right people.

The same is true of all the issues I've listed. You don't like toffs chasing about on horsebacks, or you don't like bullying activists who want to impose their view on everyone else. You don't like the sorts of people who care about blue or purple passports. You don't like Toby Young – or, conversely, you don't like self-righteous Twittermobs.

Each tribe elevates its sacred values. A policy is judged, not according to its utility, its internal consistency or its practical impact, but by how closely it accords with those values. How an MP votes on hunting or gay marriage becomes all about what sort of person he is, not about the real-world consequences of his vote. Fascinating material for behavioural psychologists, no doubt, but a strange way to run a country.