The best books on Big Projects

Could you start by defining big projects, and what kind of projects we’re talking about? Because when I started reading your book, I thought it was going to be a lot about roads and rail, but in fact, it’s a whole array of things.

There are two defining features of the projects that I was interested in. The first is that they are government projects. Strictly speaking, one of them is a parliamentary project rather than a government project, and one of them is paid for through people’s energy bills, rather than out of taxation. Nonetheless, this is about things that the government wants to do for the nation.

The other is that they’re huge. So I don’t know what the smallest project is on the list in terms of cost, but if it’s less than £10 billion, I’d be surprised. From conception to completion, projects like this usually tend to last a decade or more.

So that’s what I was focused on, because that’s what I’ve spent the last 15-20 years worrying about.

Government thinks about projects in a hugely broad way. If it is a something that’s got to be achieved, that you can’t do just by writing a regulation or publishing a guideline, then typically government will treat it as a ‘project.’ That’s why there are thousands and thousands of project managers in government.

Your book is focused on the UK. We also have quite a few readers in the US and other places around the world where people speak English. The challenges of big projects are by no means unique, but do I get the sense from your book—even though you’re not making direct comparisons—that the UK is worse at doing big projects than other countries?

I now have a few readers in other English-speaking places as well, and a few in Europe too. Almost always, if I get into conversation with them, their first response is, ‘Oh, it’s exactly like that over here as well.’ And there are, indeed, some wonderful poster children in other countries. Germans always talk about the Berlin Airport, which I don’t think they really knew they were going to finish, until suddenly it was finished many, many years late. I think the closest thing to High Speed 2 in the world is the Californian High-Speed Rail project. It’s spookily similar, in a way: announced many times with great fanfare, including by Barack Obama, when he was still the president. What they’re doing now is a bit like in the UK: it’s sort of a railway from nowhere to nowhere that you would never have built like that.

While we’re talking books, there’s a book that was the talk of liberal America last year, by Ezra Klein and Derek Thompson, called Abundance. The argument in the book is that one of the reasons why the radical right has taken hold in the United States is that the liberal left can’t do stuff. They write about the California High-Speed Rail project as an example of that. So this definitely isn’t just a British disease.

Having said that, I didn’t have to go searching for my seven or eight awful case studies. There just aren’t that many success stories for projects like this in Britain—and that is not acceptable. The comfort that other people sometimes screw it up as well is not good enough. We’re talking about something that is a chronic waste for the nation, and we can’t afford it.

So if there was one message you wanted to get across, about what can be done to do these projects successfully in the UK, what would it be?

The trains wouldn’t let me get there, so, unfortunately, I didn’t make it, but last week the All Party Parliamentary Group for Project Delivery released their first and only report. It’s got some recommendations in it, and there’s nothing wrong with them per se, but how many dozen other reports like this have we done in the past? It says that we need to do better training. There was something in it about planning regulations and more assurance, and so on.

My big message is that in order to have fewer failed projects, we need to address some of the really deep structural incentives in the system. It’s not that incapable people tried to deliver them. It’s not that they were bad ideas in the first place, although in some cases they were. It’s that all the incentives in the system encourage us to do the wrong thing.

The examples of doing the wrong thing are legion in my book, including starting before you really know what you are trying to achieve.

Are you talking about Universal Credit now?

Universal Credit is a good example. The Conservatives in 2010—and Iain Duncan Smith was the architect of it—were absolutely determined to make a fundamental change in the way that the welfare state was going to work. It had to be done. The phrase I remember somebody telling me is, ‘a year to think about it, three years to do it, and then a year to crow about it before the next election.’

It was completely unrealistic to do something as fundamental as that in that time, but they hared off at 100 miles an hour without the first idea of how they were going to do it. I think it was supposed to be done by 2017, but it will likely be 2029 before Universal Credit is fully rolled out, so more than a decade late.

Another example is HS2. Despite the fact that it had been in the pipeline for more than a decade, construction started before the project was really ready for it because there was so much pressure on the system to show progress. That’s far from the only problem with HS2, but that’s what I mean by the incentives in the system.

I hadn’t appreciated that these problems also affect the building of nuclear submarines. National security is at stake.

Absolutely. This is a subject close to my heart. I was the chair of the Submarine Delivery Agency until August 2024. This is the story of the Astute program. The Astute submarines are the attack submarines in the Navy. They’re not the ones that carry the nuclear deterrent, but they defend the nuclear deterrent and carry out a number of other vital activities for the nation.

The salient facts are that we built seven of the previous generation of attack submarines, which were called the Trafalgar class. We’ve now nearly finished building seven Astutes. The Trafalgar class boats were built in almost exactly half the time. So we’ve taken twice as long for this generation compared to the last one.

Again, as always, there are a number of reasons for that. But the one I’d like to draw attention to is that after three decades or more of building nuclear submarines, up at Barrow in Cumbria, at a drumbeat of around one every 18 months, for reasons which must have made sense at the time—to do with the end of the Cold War and the peace dividend—there was a six to seven year break between the previous class of submarines and the new class. So all of the capability for doing it dribbled away, and we had to start from scratch. The Astute program never recovered from that.

I describe this in the book in a chapter headed “The Inverse Square Law.” It’s a term I use for a quite wide-ranging problem with government projects, which is that if you take two steps away from the challenge of doing something, it looks four times easier. And if you step all the way out to Whitehall, you think, ‘What are those idiots doing? How could it possibly take this long and cost so much? Why don’t you just pull your finger out? Why don’t we get some decent people to come in and do it?’ Nuclear submarines are probably the most complicated things that have ever been built in this country and if you let that precious national capability dribble through your hands, it takes decades to recover.

And I would argue that whilst high-speed rail, or even conventional rail projects, are not nuclear submarines, they’re still not that easy. There is an argument to be had about whether it was right to stop HS2 or not. I don’t enter that argument in the book, but one thing’s for sure, having spent a decade building a delivery capability that was and still is trying to deliver a high speed rail project, if you pull the rug out from under it and let it wither on the vine, it’ll take a long time to rebuild that capability for the next railway of that scale that we decide we’re going to build. The recognition that these capabilities are hard-won, fragile and need to be looked after is not well enough appreciated.

The good news is that lots of people are still plugging away on HS2.

I felt a bit bad asking you for one takeaway because, by definition, these projects are incredibly complex.

Well, another takeaway would be not to allow them to be so complex. The problems that we try to solve with projects—whether it’s national defence or levelling up or economic development or decarbonizing the country—are incredibly complex. There is no getting away from that. But don’t allow your project to become incredibly complex—something that’s never done before, with no supply chain that knows how to do it, or getting different parts of the economy to work together in completely different ways, or changing the behaviour of the population so they do something different. If you make it complex like that, don’t expect it to work because it won’t. You have to address complex problems through simpler projects.

That makes sense.

It makes sense to you. I wish it made more sense in Whitehall.

Let’s go through the books you’ve recommended.

I’m not suggesting anyone go out and read them all; it’s probably 1,500 pages or so. But this is an under-explored subject and these are interesting to dip into.

Let’s start with The Blunders of Our Governments by Anthony King and Ivor Crewe. What’s this about, and is it covering similar ground to your book?

This was published in 2013, so a lot of it is previous generation. It’s the canonical text in this area in Britain. It’s a very good book. It’s covering things like the poll tax, the Millennium Dome, the ID card scheme and Metronet, which people will have forgotten about but was a huge privatization of the London Underground engineering.

In many ways, it covers similar ground to my book, although their focus is more on the policy flaws than on the delivery challenges. It’s also an outsider’s book, whereas mine is an insider’s book, which is unusual.

What qualifies as a policy flaw?

The biggest example is the poll tax. It was a flawed policy that the Thatcher government held on to long past the evidence saying that it was going to be a disaster. They were going to get rid of the rates and put in place a local tax based on how many people lived in a house. That’s naturally regressive, but they thought it would be okay, because the amount of money would be quite low and nobody would be getting huge hikes in their bills.

In fact, what actually happened was that people got huge spikes in their bills. There was a revolt, really, and it was one of the things that cost Margaret Thatcher her job. John Major quickly scrapped it. The reason the poll tax was a disaster wasn’t because they couldn’t build the IT system. The problem there was the policy itself rather than the practicalities of putting it into operation.

If I have one criticism of this book—which is true of a number of these kinds of books—it’s that it’s far too long: well over 500 pages. One of the best things that my publisher, Emerald, did was to give me what I thought at the time was a very mean word count. The Delivery Gap is about 170 pages. Consequently, I think quite a lot of people who started it finished it.

Let’s go on to How Westminster Works… and Why It Doesn’t by Ian Dunt. Tell me about this one.

This came out in 2023. It’s a good read, written in a really engaging style. It’s kind of an outsider’s guide to how the British government actually works. What do ministers do? What do civil servants do? What do backbenchers do? What happens in the House of Lords? What’s a select committee? It’s a very instructive book, but told in a very engaging way, and focused not just on how things work, but what the impact of the way they work is.

The reason I put it on this list, Sophie, was for its opening chapter. He starts off with the story of a minister in the Cameron government, Chris Grayling, who took over as the Justice Secretary and—a little bit like the Universal Credit story—launched into an ideologically motivated privatization of the probation service. It’s the idea that ‘all these civil servants in the probation service can’t be doing a good job, if we put it into the private sector, it’s bound to be better.’

It was an unmitigated disaster. It took many years to unpick, at the cost of major societal damage. But he tells that story absolutely brilliantly. It inspired me to work really hard to tell these quite complicated stories in a way that the general reader would understand.

One thing he says that seems to overlap with a point you make in your book is the quick turnover of ministers and civil servants, and how knowledge is lost over time.

I don’t think we should expect our ministers to come in full of knowledge. We don’t want them to run the projects. We want them to make the big decisions. But we have this fast turnaround in the departmental officials too.

It’s not just about knowledge. These projects, typically, for a short one, take a decade from the big first decision to do something, to actually finishing it. And if you have—charitably—five ministers in that time, but it could be eight, and you have four or five delivery leads in the government department, no one gets to know each other well enough to trust each other. So the relationships are never strong enough to build the momentum to carry you through difficult times.

So yes, he’s absolutely right about that, and I’m sure that it isn’t only major projects that suffer from this problem.

Let’s go on to How Big Things Get Done (2023) by Bent Flyvbjerg and Dan Gardner. This book is not about the UK but international in scope.

Bent Flyvbjerg is one of the world’s major project gurus. He’s been writing books on mega projects for a couple of decades. This was his attempt, working with a Canadian journalist, to write something a bit less academic and easier to get your head around.

It’s not talking just about government projects in the way that I do. It’s probably more focused on physical asset projects—bridges and buildings and roads and railways etc.  It’s a good read,  and even though it’s 300 pages long, quite a fast read. It came out about three years ago and one or two of his recipes for different ways of thinking about big projects have already become part of the language in the project world.

There were two that stuck with me. One is ‘plan slow and act fast.’ What he means by that is along the lines of what we were saying earlier: Don’t start till you really know what you’re doing. It does take a surprisingly long time to really, really work out what a project is all about. If you try to skip that part, or rush it because of the pressure to be acting—to be buying land or digging holes or signing contracts—then you suffer in the long term. It’s a well-known project management saying: the sooner you start, the later you’ll finish. You need to devote the right amount of time to the planning in order that you can do a really fast job of the building, which is the expensive bit.

The other is ‘think right to left.’ This is project management stuff. Normally, when you plan, you think, ‘What are we going to do first, and what will we do next?’ That’s left-to-right thinking. Right-to-left thinking is, ‘This is where we want to end up.’ You then go backwards to what you have to do first.

There are dangers with right-to-left thinking—all of these truths have counter-truths to them: that’s the life of project management—but starting with the end in mind is really good advice.

So it’s a good, quick read. It’s from an academic point of view, but not written in an academic way.

Let’s take a look at the next book, which is  Conundrum: Why Every Government Gets Things Wrong – and What We Can Do About it.

This is by Richard Bacon, who was a member of parliament. As an MP, he sat on something called the Public Accounts Committee (PAC) for many years. It sounds terribly boring, but if you work in Whitehall, you live in fear of the Public Accounts Committee. They can call in ministers, officials, someone from the arm’s-length delivery body, and grill them about why this has taken twice as long as you said it would. Senior civil servants will literally lock themselves away for three days before a PAC meeting and practise the answers to all these difficult questions.

This is his book, written with Christopher Hope, a journalist, about ‘What are all the lessons from this?’ It covers similar ground to The Blunders of Our Governments and over a similar chronological period.

What is interesting is that it comes from the perspective of an MP, and therefore somebody who does know a bit about politics. An awful lot of people will propose solutions to the problems of delivering these great big projects which work perfectly in the minds of a technocrat, but which just aren’t going to work politically. I think Bacon is less likely to fall into that trap than some of the others.

There’s a lot in the book about IT projects. The big failures of IT projects tend not to be that costly. There are some exceptions, but you don’t need thousands of people in overalls digging holes to do IT projects. So they tend not to be all that expensive, but they can still represent a great waste of governmental time and effort and have a big negative impact on the citizen.

This isn’t a book I would recommend anyone go searching for. If the website was four books, I probably wouldn’t have had this one on the list. But there really aren’t a lot of books written on this interesting subject of how to deliver great big things for the nation.

When you say that people think about solutions or have ideas that just aren’t viable politically, what’s an example of that?

Let me give you one. A long time ago, around 2008, I used to run a part of the Treasury that focused on how to improve these big projects. Because of this business that if you start the project badly, it will never recover, we invented what we called a ‘starting gate.’ Before you launch the project, before the minister stands up and says, ‘We’re going to do this, and it’s going to cost this much, and it’ll be finished by this date, and it’s going to have all of these benefits’, you must do an assurance review to unearth whether you really know what you’re talking about.

After I left, the Labour Party lost the election and David Cameron’s government came in. Cameron wrote to all the secretaries of state in Whitehall and said that there must be no announcements of projects from now on that hadn’t been through a starting gate. He wrote to everybody about this, together with a list of other things that were technocratically good ideas.

You cannot stop a politician who is in a hole and needs to burnish his reputation after three days of bad news stories about the state of infrastructure from standing up and saying, ‘I’m going to sort this out. We’re going to have a project.’ It sounds great from a technocratic point of view, but you just can’t. You have to acknowledge the reality that we live in a political world and find a way to help the minister to say something which is not a hostage to fortune and which may not get quite as many short-term political brownie points—but avoids everybody looking really, really stupid in five years’ time.

We’ve got to your last book, Clashing Agendas: Inside the Welfare Trap by David Freud. This focuses on Universal Credit in particular.

I really did enjoy reading this book. David Freud was an investment banker by career but was plucked into the Tory government in 2010. He was stuck in the House of Lords so he could be a minister, and was Iain Duncan Smith’s wingman, if you like. He tells the story of the Universal Credit program, and it’s the only one of these books that gives the voice from the inside, from the ministerial point of view.

It must be terribly hard to write a really frank and honest book from the inside of something that went so badly, at least for many years, but he’s done a really good job. You have to care about the subject, which is a bit technical, but it’s readable. He talks about some of the blind alleys that they went down, and the difficult political fights, particularly with the Treasury (George Osborne and Iain Duncan Smith were not particularly good friends, if I can put it like that). It also went through some near-death experiences from the technology point of view, of just not being able to get it to work.

Universal Credit was dancing on the edge of being canceled many times. And if that had happened, we’d have lost five years. It’s been a dreadful project in terms of its schedule. It’s been a dreadful project anyway. But we’d have had to go back to square one, and a new government would have had to start again.

So that’s the story that David Freud tells quite frankly.

And has Universal Credit solved the problem of the welfare system, so that it doesn’t penalise you for working?

That’s a really good question, and one that I end the chapter on because plainly it hasn’t. If one is to believe what one reads in the newspaper, we are now in a situation which is exactly the one that Universal Credit was intended to solve. Being in work should always be better than not being in work, and there should be a heavy incentive for everybody who can to work.

The way David Freud describes it, the Treasury (i.e. the Chancellor, George Osborne) wanted to take the benefits of Universal Credit—lower costs because everybody would be in work and paying tax—in welfare cuts before the upfront work of getting Universal Credit in and generating its magic came to fruition. Whether that’s true or not is too difficult a question for me. It’s not really my specialist subject.

It ultimately led to the resignation of Iain Duncan Smith—although the Brexit referendum, where he was on the wrong side of the government, might have had something to do with it as well.

There was a critical need to simplify and rationalize the benefit structure. It was a thicket of historical allowances and benefits and so on. It was a mess. Everyone knew it was a mess, and the idea of trying to sort that out was not a bad idea. But the ideological politics behind it were, ‘We’ll get rid of all of those benefits, and we’ll structure benefits so that they always incentivize work.’ It’s not evident that that’s been a success.

We’ve discussed policy versus delivery. In the book, I think you suggest that in decision-making, not enough attention is given to the delivery. Could you say a bit about that?

There is a longstanding concern that the civil service is staffed with highly intellectual generalists. Harold Wilson’s government was concerned about this, and he had people advising him on it. You get on in the civil service through your intellect and your ability to devise and sell policy, rather than to deliver solutions.

It’s often described as a lack of parity of esteem. It’s not a criticism of policy professionals. But if you join the civil service as a fast-stream, generalist policy person, you might expect to be sitting in front of a minister within three or four years, talking to them about something—welfare policy, or whatever it might be. If you join the civil service from university in their project management stream, you might never meet a minister in the whole of your career.

So the power rests with policy and announcing policy is very, very cheap. Delivering huge projects takes a very, very long time and costs a great deal of money. The immediate payoff is the announcement.

I hope I don’t come over in this book as saying ‘I’ve got these simple three things, if only we do these, everything will be okay’ because it isn’t that kind of a world. But one thing I felt increasingly stronger about as I wrote the book is that we have to find a way of separating announcements from actual investment decisions. As I said before, ministers will always make announcements. That’s the nature of the beast.

So when a minister makes an announcement about a project to assembled journalists, rather than them rushing to tomorrow’s paper and writing ‘Government to Build High Speed 3’, the question I’d like them to ask is, ‘Has the business case been approved? Has the formal governmental decision to spend this money been taken yet?’ And the honest answer, at that stage, will be no. And then their question could be, ‘When will that be and can we talk about it again then?’ We need to shift the focus from political announcements and statements of policy direction to formal investment decisions. Then we can hold people to account for them. You can’t hold somebody accountable for what a minister might have said in 2014. We have to build more candour into this world.