Corbyn didn't happen in a vacuum - we all had a role to play

Flicking through the index of my diaries for 2003-2005, I note that Theresa May's name appears once, and Jeremy Corbyn's not at all. I say that not to make a cheap shot at either, but to illustrate how much our political scene has changed since those days when, even despite the Iraq War, New Labour was so dominant in the UK landscape we headed to a third successive election victory.

Had I predicted then that by the time the diaries were published just over a decade later Corbyn would be Labour leader amid talk of the party splitting, May prime minister on the back of Britain voting to come out of the EU, and Ed Balls a contestant on Strictly Come Dancing, I fear my doctor would have asked me to up my medication.

But looking back it is possible to see some of the seeds being sown for the situation in which we now find ourselves. Like the steady reminders coming in from Philip Gould's focus groups and from MPs in traditional working class seats that immigration was becoming an ever bigger problem, and being mixed in the public consciousness with asylum, crime, pressure on public services and Europe. We saw the problem, but struggled with the solution.

On Europe, the clear sense that despite Tony Blair's best efforts, we never quite shifted the dial sufficient for British people to feel, as the French and the Germans tend to, that the EU was a force for good in their lives. We see, not least in the ongoing arguments I was having with the right-wing press, a media that was continuing to portray Europe and the problems for which it was blamed in a relentlessly negative light. We see a Tory party seeking to exploit rather than challenge it, which would set up problems for David Cameron when it came to this year's referendum – he discovered that a few weeks of fighting for Britain in Europe was insufficient to push back years of scepticism he had failed to challenge.

Then there is the ongoing impact of what became known as the TBGBs – the difficult and often tortured relationship between Tony Blair and Gordon Brown. Two great talents in so many ways, but the sheer amount of time and energy that was taken in dealing with the fallout of their fallout was without doubt a debilitating factor, and one with lasting consequences.

Though I had left my official role in Downing Street at the opening of this volume of diaries, the title, Outside, Inside, indicates that I remained closely involved. Indeed at one point the title was due to be Never Really Left but I felt in these Corbynista times that would be open to fairly obvious misinterpretation.

One of the roles I had was to help TB put together the strategy for the election campaign. We wanted to fight it on the economy. Yet the chancellor, Gordon Brown, was reluctant to get involved in campaign planning or in committing to a role we drew up for him. TB, fairly regularly, talked of the "nuclear option" – sacking him. But there was a reason he called it "nuclear" – the fear they would both be destroyed by that course of action.

So, in the end, he decided not to, and that meant we had to push the boat out to get Gordon back in. I was in charge of putting together yet another so-called "deal" under which Tony made clear that Gordon would be staying as chancellor and that when the time came he would back him to take over as prime minister. Both Peter Mandelson and Cherie Blair felt I went far too far in ceding so much to Gordon and they may have a point. But we were running out of time to get a proper campaign strategy and structure in place.

The tragedy is that when it came to the campaign they worked together really well and many times I wondered why it couldn't always be like that. But the moment the election was over, as the next volume makes clear, the old battles resumed.

One of the side effects of all this is that other political talent got squeezed out. So focused were we on seeking to resolve the TBGBs that despite all the doubts we had all our energies went on trying to make sure they could work together for the campaign, and the price of that was a GB takeover.

There are plenty of other names in the index who might have emerged as possible leaders and PMs – John Reid, David Miliband, Alan Johnson, Charles Clarke, Alan Milburn, Robin Cook – but the combination of GB's political muscle and TB's non-nuclear decision meant their chances were stunted.

So many of the best talents we had have left the scene, some to election defeat, some to death, some just ground down and opting for a different sort of life.

So Gordon became leader without a challenge and when he went, Ed Miliband won the leadership and by the time it came to choose his successor, the sense was there was not much to choose from, and amid the desire for "something different" Jeremy Corbyn won. And now that he has won again, we have to accept that New Labour may have succeeded in much, but we failed in cementing the Blairite legacy in the same way as the Tories' three times election winner Margaret Thatcher did on the other side.

The Tories still feel they have to define themselves to some extent according to the Thatcher agenda. A succession of Labour leaders, starting gently with Gordon, more obviously with Ed Miliband, and now rampantly under Corbyn, have sought to define themselves as "not Blair" and "not New Labour".

In one meeting I had with Gordon at his home in Fife, in late March 2005, he said that Tony's choice agenda in public services was leading one of two ways – to future Labour leaders rolling it back as the only way of holding the party together, or a future Tory government going full steam ahead to genuine privatisation. Maybe we are ending up with both.

By the time Gordon had lost to David Cameron, and Ed Miliband took over, Ed was even more strongly of the view that a distancing from the Blair era was required for electoral success. Unfortunately, that extended to his ventilating rather than challenging the Tory attack on Labour for having created "the mess we inherited", another strategic error whose seeds were sewn earlier when Ed was a regular caller to my office making the case for Tony's departure and a radical shift of approach away from New Labour.

Tony always thought New Labour was the answer. He still does. Gordon always felt that he needed to define himself against Tony. Ed felt he needed to distance himself from both. Tony begat Gordon begat Ed begat Jeremy. Corbyn didn't happen in a vacuum.

And as the Tory conference begins, if you were to ask Mrs May to rate those four in order of electoral threat at the time of their leadership, I suspect that is the order in which she would list them. But all of us have played a part in getting to where we are.

Outside, Inside is published by Biteback Publishing and will be available from ‪6 October, £25