The civil service desperately needs reform - and it can start by getting political

Remainers and Brexiteers briefly played football together on Christmas Day in No Man's Land. Thereafter, hostilities resumed. The first casualty has been Sir Ivan Rogers, our man in Brussels. Remainers would like to give him a state funeral. Brexiteers want him buried in an unmarked grave.

There is a perfectly respectable case for Sir Ivan's departure. He was quite right to say that with Brexit negotiations imminent, it would be better to have someone in post who could see them through, rather than someone scheduled to depart in November. And even his notorious email had some justification. Viewed from a purely internal perspective, it was a perfectly reasonable pep talk to his beleaguered troops as he was about to leave the field.

But I have found myself becoming less sympathetic to Sir Ivan's plight. Tim Shipman's excellent book on the referendum All Out War (a geek must-read) points out that he has threatened to resign many times before. And he will have known full well his email would be published, causing an uncomfortable moment for the government. There was quite clearly some method being employed here.

Theresa May and her team have handled the fallout very well. Sir Ivan has been replaced by Sir Tim Barrow, and troubled waters have been smoothed over. There was briefly talk of appointing an outsider, and there is still talk of an ideological purge of the civil service (who can now doubt that we are back in the seventeenth century in some respects?).

As a Minister, I worked with officials who ranged from the excellent through to the mediocre and the plain awful. But I never doubted their commitment to me, and what I wanted to achieve. I never doubted the impartial nature of their advice, and never once came across any hostility or politics. I know other Ministers did not feel the same, whether in the Cabinet Office, or the Department for Education. Perhaps I was one of the lucky ones.

Nevertheless, as a junior minister, I keenly felt the lack of political advisers. Modern Whitehall is vast, and the workload is huge. Ministers can quickly feel isolated. They have little or no say in which officials will lead on particular policy areas, and it is difficult to get outside expertise in quickly. While civil servants may be impartial, they also sometimes lack the ability to understand political priorities, or to share the sense of political urgency.

For years, politicians have run shy of increasing political appointments. The annual special adviser bill is an easy off-the-shelf news story. The smaller it is, the more virtuous the government. But the fact remains we should have many more political appointments.

As the US changes president, we are reminded that the White House administration appoints 4000 officials, from ambassadors to ministers. In Whitehall, there cannot be more than a hundred political appointments. This government has a huge agenda, not just Brexit, but social reform and other priorities such as housing. Ministers would benefit hugely from additional outside expertise, and many more political advisers are prepared to help them stay on top of their agenda.

There are problems with this thesis. Ministers come and go, and one Minister's appointment may not be to the liking of another. It's a fine thesis to have more political appointments when you are in charge, but what about when the other lot get in. It would undermine the role and morale of career civil servants who expect to get to the top. Appointing senior advisers could also end up undermining the authority of the minister.

But these are quibbles. We have a Victorian structure in Whitehall which has barely changed in 150 years (tell me what a Grade 7 is ...). Reforming the civil service would be a good place to start in setting this country up for a bright future.

The Brexiteers are uncompromising Tory modernisers. Douglas Carswell writes about i-democracy; Dan Hannan campaigns for global free trade; Michael Gove has set out a humane and liberal programme for prison reform; Boris Johnson champions the City and multi-culturalism. It's time for them to remember their modernising roots, and campaign for modernising our civil service.

Ed Vaizey is Conservative MP for Didcot and Wantage and was Minister for Culture & Digital Economy 2010-16