IT projects in the public sector: Lessons from practice

Increasing Dependence on IT within the Government

Each year, countless IT projects are launched within the government. "IT projects" is an umbrella term that includes ICT, IT, projects, and programs. These projects range from small improvement initiatives to large-scale transformations that completely overhaul existing ways of working. In 2022, 35 new large IT projects were launched with a value of more than 5 million euros (Hartholt, 2023). These IT projects often promise faster service delivery, lower costs, and more efficient processes. The reality, however, is different.

Programmatic Approach Fails in Practice

Although a programmatic approach within the government is often seen as a proven method to realize large IT projects, its success in practice is questionable. Large-scale IT projects repeatedly get out of hand. Research by AG Connect and Binnenlands Bestuur shows that the total cost of IT projects within the government rose to €6 billion in 2022—€1.3 billion more than budgeted—with overruns of up to €700 million in the ten largest projects. In addition, there are years-long delays, and many change initiatives do not function as promised. Notorious failures are still fresh in our memory, such as the ICT for the Environment Act, the Tax Authority, and the ERP system of the Ministry of Defence.

The ‘Fat Tail’ Distribution of Cost Overruns

This is further confirmed by Bent Flyvbjerg. Flyvbjerg states that IT projects follow a so-called “fat tail” distribution. This means that most projects have small cost overruns, but there is a small number of projects with extremely large overruns. His research shows that 18 percent of IT projects have cost overruns of more than 50 percent, measured in real terms. Strikingly, for these projects, the average overrun amounts to a staggering 447 percent. This figure illustrates the severity of the overruns in the so-called “tail” of the distribution. In short, IT projects follow a power law distribution: most overruns are small, but a small number of projects have extremely large cost overruns.

The Need to Dig Deeper Structurally

This pattern of failure is not an incident, but a structural problem. The issue lies in the way these projects are set up, planned, and executed. Despite the many evaluations and studies, the question of why these projects repeatedly fail remains unanswered. In this article, I aim to investigate the deeper causes of this failure and offer a solution.

The Empirical Reality of IT Project Cost Overruns: Discovering A Power-Law Distribution. Bent Flyvbjerg 2022.

The Real Causes of Failing IT Projects

In their books How Big Things Get Done and De Bodemloze Put, respectively Bent Flyvbjerg & Dan Gardner and Niels Groen explain why large-scale (IT) projects fail and how they can be successfully managed. They reveal that it’s not the technology that fails, but the way these projects are set up, planned, and executed.

Two of the main causes of failing (IT) projects are deliberate deception and structural optimism.
Deliberate deception refers to intentionally underestimating costs, setting overly short timelines, and exaggerating the benefits of a project in order to gain approval. In doing so, managers and project leaders create an artificially positive image, while real risks and costs are downplayed. This can result in projects becoming unmanageable once reality catches up. The root of this deception lies in the desire to obtain approval—so the very foundation of the project is flawed. From the beginning, the aim is often to get unrealistic plans approved, leaving no solid basis for the project's success. Even when the situation deteriorates, teams tend to cling to the original plan—often because they are locked into the initial approval and fear having to revise or cancel it.

Alongside this deception, there’s structural optimism: a psychological trap in which people overestimate their capabilities and underestimate the complexity of a project. Flyvbjerg calls this the “everything-goes-according-to-plan principle”—the illusion that once there is a tight plan, everything will go smoothly. But reality tells a different story. According to Flyvbjerg’s research, 91.5% of megaprojects are delivered late, over budget (or both), and 99.5% deliver less value than promised. This means that virtually every large project fails to meet its original goals. Yet, even as signs of failure pile up, people persist with the original plan. As the Joker in The Dark Knight aptly puts it: “Nobody panics when things go according to plan. Even if the plan is horrifying.”

The Trap of Political Wishful Thinking and Escalating Commitment

This problem originates in the planning phase. Many projects begin with overly ambitious goals and an unrealistically rosy outlook. Instead of drafting a realistic plan, proposals are presented that are politically desirable but practically unfeasible. When the plan starts to fail, there is no course correction—instead, people remain committed to the original path. This is a classic case of escalating commitment, as described by Groen in De Bodemloze Put.

Groen describes that in failing public IT projects, two key dynamics frequently occur: an ineffective course of action, and escalating commitment to that same ineffective course by those in charge. He identifies five contingencies that lead to an ineffective course, and five that cause escalating commitment. Contingencies are events or conditions that may occur, but aren’t guaranteed.

What emerges is that almost all public IT projects—even the successful ones—face at least one contingency that leads to ineffectiveness. However, it’s the projects that also encounter contingencies causing escalating commitment that truly spiral out of control. It’s crucial to understand that an ineffective course often stems from a flawed project setup. Escalating commitment arises when the signs of that ineffectiveness are ignored or suppressed.

Programmatic Working as a False Solution

For this reason, large IT projects within the government are often set up as programs. In theory, a programmatic approach should ensure coherence, control, and risk management—but in practice, it’s far from a guarantee of success. Programs are designed on such a large scale that they end up paralyzing themselves, with complex governance structures and slow decision-making as a result. Instead of working adaptively and iteratively, rigid plans are followed—plans that are often outdated by the time they are implemented. This lack of flexibility during execution causes projects to stall when reality no longer matches the original planning.

What further undermines these projects is the persistent misconception that failing IT projects are caused by technical complexity. This is incorrect. While technical complexity is a complicating factor, the success of IT implementations is not determined by technology alone, but by a complex and unpredictable interplay between technology, people, and organizational and institutional structures. The real issue is, therefore, organizational. Or as Flyvbjerg puts it: “Projects don’t go wrong, they start wrong.” The government invests billions in software and systems but overlooks the organization’s capacity for change. New technologies are implemented without sufficient attention to the people, processes, and structures needed to make them work successfully. This creates a gap between technological development and organizational reality. Technological improvements may be prioritized, but the organizational capacity to actually realize them is lacking. As a result, projects stall and investments in digital innovation yield little return.

Where Execution Goes Wrong

In addition to mistakes in the design and planning phases of these projects, many things also go wrong during execution. Too many projects get stuck in slow decision-making, poor collaboration, and unclear responsibilities. Often, it’s only during execution that it becomes apparent that the plans made during setup are unfeasible, or that stakeholders lack the necessary resources or knowledge to achieve the project’s goals. Execution is where planning failures become most visible.

One of the biggest execution issues is the lack of flexibility in how projects are carried out. Instead of using an iterative approach—where adjustments can be made based on interim results—many projects remain stuck in rigid structures that are incapable of scaling up, adjusting, or shifting course when problems arise.

Inefficient governance is also a critical problem during execution. Instead of enabling fast decision-making and course correction, key decisions are often delayed by bureaucratic layers and a lack of clear mandates. Projects get stuck in what's called a “governance trap,” where too many people have to be involved in final decisions, hindering progress.

Another major issue is the lack of collaboration between stakeholders. During execution, it often becomes clear that the interests of different departments are poorly aligned, leading to misunderstandings, delays, and inefficiencies. This problem also stems from the project’s original setup: if responsibilities and mandates weren’t clearly defined from the start, then roles during execution are often poorly fulfilled.

Overload and Lack of Focus

In addition, there is overload due to too many projects being initiated at the same time. Research by Turner [1] shows that public organizations structurally undertake too many change initiatives simultaneously, leading to a lack of focus and unclear priorities. Government initiatives lack the decisiveness to bring about real change because resources are spread too thinly and priorities shift whenever a new initiative appears on the agenda. Instead of making sharper choices, ambitions are stacked, and the organization becomes overburdened.

Time for Fundamental Change

The pattern is clear: government IT projects do not fail due to bad luck, but because of structural flaws in setup, planning, and execution. The government must stop planning too optimistically and instead base budgets and timelines on data rather than political desires. This is not only a matter of cost and time—it’s about the organizational capacity to actually achieve change.

Moreover, the fear of shutting down failing projects must be replaced by a more realistic approach: better a hard decision now than a bottomless pit later. Most importantly, the focus shouldn’t be solely on implementing technological improvements, but also on transforming the organization itself. Without the ability to change, any technological progress remains an empty promise.

The hard truth is that government IT projects don’t spiral out of control by accident—they are designed in such a way that failure becomes inevitable. As long as we continue down the current path, billions of euros will keep vanishing into bottomless pits. These structural flaws are not an abstract issue—they happen in real life and lead to large-scale waste. A recent case study shows exactly how these problems manifest in practice.

[1] https://www.consultancy.nl/nieuws/36978/in-hoeverre-zijn-publieke-organisaties-in-staat-te-veranderen

Promising Plan, But an Organizational Minefield

A large ongoing program within the government.

A major government program was launched with the aim of improving a complex and essential system in the public sector. While technically challenging, the program is manageable. The individual technical solutions are relatively simple and feasible—if they were to be implemented separately. However, the setup, structure, and organizational context in which the program takes place are highly complex. This makes it a textbook example of the gap between technical development and organizational reality.

As is often the case, it all began with grand promises. In this instance, the Dutch House of Representatives was convinced by a carefully crafted plan that broadly outlined the program’s objectives. Believing that a programmatic approach would offer control over the complexity, several ambitious sub-programs were launched as part of a wider policy agenda. Funding came from a ministry, while execution was spread across multiple implementation agencies. Each sub-program had its own focus, budget, and timeline. The program plans looked flawless on paper—a tight schedule was laid out, and the rollout appeared to be a simple matter of following the plan. On paper, everything seemed to make sense.

Delays Due to Lack of Cooperation and Clear Steering

But once the program got underway, it became bogged down in a swamp of conflicting interests, sluggish governance, and poor coordination. The change tasks of the various implementation agencies differed significantly. This fundamental disconnect was ignored in the early phase, based on the optimistic belief that “good collaboration” would solve everything. Yet without a clear shared direction and firm agreements, even relatively simple tasks—like creating a unified architecture model or setting standards for privacy and information security—turned into lengthy and frustrating negotiations. Additionally, thinking remained siloed, both between and within the agencies, which further complicated any cross-organizational approach.

The governance structure—meant to bring order and control—instead became an obstacle to progress. The ministry steered on abstract policy goals, making it difficult to translate them into actionable steps. This led to endless discussions rather than forward movement. Within the program itself, there was a lack of decisiveness. The steering committee, which was supposed to make key decisions, turned out to be a paper tiger. Committee members lacked proper mandates, no firm decisions were made, and no one felt ultimately responsible for the program’s progress. On top of that, there were several overarching programs and domain-specific projects with strong interdependencies and their own steering structures and governance systems. This increased complexity and blurred responsibilities, further undermining efficiency and making it difficult to realize shared goals.

Overly Optimistic Reports Mask Deep Structural Issues

The everything-goes-according-to-plan principle prevents intervention when the program gets stuck. Official progress reports remain optimistic: “challenges are being addressed,” “discussions are ongoing,” “solutions are being explored.” Only after a year is it acknowledged that the program faces structural problems. By then, millions have already been spent without any tangible results. It takes two full years before improvements are made to the governance structure.

The program plans also provided little guidance for execution. The scope was unclear in relation to other programs within the implementing organizations. Endless debates followed over what the program should deliver, but there was no clear path forward. This ambiguity in scope caused further delays, as nothing could be implemented yet.

Rigid Execution Prevents Course Correction

At one of the implementing organizations, an iterative approach was lacking. The program would benefit from a flexible, step-by-step working method, where interim results could be used to adjust course. But that approach was absent. The program got stuck in rigid structures, making it difficult to implement changes when the first problems arose.

The consequences were predictable. The program remained in its start-up phase for too long, with no clear priorities or direction. Timelines stayed under pressure, and it became increasingly unlikely that the original goals would be achieved within the planned timeframe. Meanwhile, costs continued to rise, while the promised results failed to materialize.

A Symptom of a Structural Problem

This case is no exception. It reflects a recurring pattern in large government IT projects. Organizational complexity—combined with structural flaws in setup, planning, and execution—is the biggest bottleneck. This example demonstrates that understanding and managing the organizational context, and carefully designing the program structure, is just as crucial as the technological improvements themselves.

IT projects are complex, high-risk, and fat-tailed, which makes it essential to carefully consider how to approach them in advance. This requires a realistic approach to budgets, timelines, and promises, without falling into the trap of optimism bias or deliberate deception. Base your plans on data from previous large and comparable projects, using reference class forecasting to help temper expectations. In addition, it is crucial to take mitigating measures for the greatest risks inherent in IT projects. Groen has identified five contingencies that can lead to an ineffective course: technical complexity, adoption problems, conflicts between stakeholders, communication and coordination issues, and unclear or changing requirements. To address these risks effectively, an organizational structure must be set up that can detect and mitigate these contingencies early. You will always face at least one of these contingencies; the key is to prevent them from escalating and leading to excessive commitment to a failing course, thereby ending up in the tail of the distribution.

To address these challenges, we introduce the concept of 'Adaptive Flow' (AF). An iterative approach that helps gain control over complex IT projects by continuously aligning strategy, execution, and changing circumstances. AF is not a theoretical model, but a pragmatic way of working that prevents IT projects from falling into the traps of the bottomless pit, structural optimism, and deliberate deception. Where traditional project approaches are rigid and inflexible, AF creates movement, coherence, and agility. It forces organizations to sharply define what truly has priority.

AF addresses the structural organizational problems of large IT projects in five clear steps.

An IT project without clearly defined goals is like a ship without a compass. AF forces organizations to formulate realistic, measurable, and achievable objectives. This prevents programs from being drawn into a bottomless pit due to vague ambitions and unrealistic expectations. Thinking from right to left supports this process by starting with the end goal in mind and working backwards toward the necessary steps, creating a clear and feasible path to success. By setting concrete goals, the trap of unrealistic expectations—often at the root of failing IT projects—is also eliminated, as previously discussed.

Conflicting priorities are the death blow for any IT project. AF ensures that all involved parties—from policymakers to implementers—move in the same direction. This prevents the paralyzing situation in which organizations get stuck in endless coordination, as seen in the case study. By creating a shared framework, energy is focused on the most urgent goals. AF helps prevent IT projects from stalling in endless alignment without making progress.

Where IT projects often assume a perfectly mapped-out path, AF recognizes that reality is more unruly. A joint, iterative planning process brings focus and clarity. This prevents IT projects from becoming a collection of separate initiatives, each driven by its own agenda. A planning process that makes progress tangible and encourages collaboration replaces chaos with direction. The principle of “Think Slow, Act Fast” supports this approach: first, careful and thorough planning (thinking in slow motion), identifying risks and obstacles. Then, rapid and focused development and testing (acting fast). It’s not just about drafting a plan on paper, but about continuous iteration, testing, and learning before rolling out the project at full scale. This creates the flexibility that is often lacking in traditional projects, where rigid plans hinder execution. It provides a solid foundation for implementation, allowing for quick course correction without getting stuck in excessive detail planning.

Technical issues often receive much attention, but the real problems lie within the organization. Bureaucratic blockages, lack of support, and political pressure are too often left unaddressed. AF forces organizations to make these organizational obstacles explicit and to address them structurally. This prevents projects from stalling in the molasses of slow decision-making, where responsibility for problems is shifted rather than solved. By using a modular, Lego-like approach, taking iterative and incremental steps, value can be delivered in smaller, manageable pieces. This makes problems visible earlier and allows them to be resolved more quickly—without the entire project getting stuck. Instead of waiting for the “big solution,” progress is made continuously, enabling quicker adaptation to changing conditions and greater flexibility during execution.

Too many programs continue to muddle through without a clear benchmark for success. AF introduces a process of continuous progress validation, where results are not only visible at the end but monitored and adjusted in short cycles. This aligns with lessons from How Big Things Get Done and De Bodemloze Put, which emphasize that realistic planning, iteration, and timely intervention are essential. Problems are solved before they escalate, and IT projects remain adaptive and agile. Through strict progress validation and iterative learning, AF prevents projects from falling into the “everything-goes-according-to-plan” mindset, which typically leads to delays and cost overruns—as seen in real-life examples of failed IT projects.

What makes AF unique is that it treats technical improvements and organizational complexity as one integrated whole. It emphasizes the how, enabling programs not only to formulate goals but also to organize the path toward them effectively. Instead of relying on static plans that quickly become outdated, AF offers a dynamic framework that adapts to reality.

In a political-administrative context—where projects are often pressured by shifting priorities and political interests—AF makes progress transparent. Policymakers gain better insight into the actual status of IT projects and the bottlenecks that need to be resolved. This prevents projects from silently stalling for years while the outside world remains under the illusion that “everything is going according to plan.”

This approach helps avoid IT projects repeatedly falling into the same fundamental traps. AF provides control over complexity, clarifies priorities, and creates an iterative process in which organizations not only learn but also improve continuously in handling change. This increases the likelihood of success, enables more efficient use of resources, and keeps expectations more manageable.

The failure of government IT projects is rarely due to shortcomings in technological improvements. The real challenge lies in the surrounding organization. Structural optimism, unrealistic planning, poor risk detection, and sluggish governance cause IT projects to stall before they even get off the ground. The solution does not lie in sticking to fixed plans, but in the adaptive capacity to continuously adjust course and to give organizational complexity the same weight as technological issues.

By detecting problems early and planning execution in an iterative and realistic way, Adaptive Flow (AF) offers a pragmatic solution. It avoids the pitfalls of structural optimism and deliberate misrepresentation, and makes programs more agile and effective. AF increases transparency in progress, helps set realistic expectations for outcomes, and enables more efficient use of resources. Instead of getting stuck in unrealistic plans, AF allows organizations to respond flexibly to a changing reality and to make measurable progress.

Success doesn’t come from following a perfect plan, but from the capacity to continuously anticipate and adapt. Only by balancing technological improvements with organizational reality can government IT projects truly make an impact. Because one thing is certain: muddling through is not a strategy—it’s a recipe for failure.