Difficult problems

In motivating Agile and when discussing with people skeptical of Agile, I believe it is always important to clearly distinguish between complex and complicated problems as defined by the Cynefin framework. Just in case you are not familiar with Cynefin, to put it in very simple terms, complicated problems have a fairly predictable cause->effect relationship and the same course of action that has worked once (“best practice”) will usually lead to nearly the same result again and again. In complex problems on the other hand, the relationship between cause and effect clearly exists (otherwise, they would be a chaotic problems), but it can only be deduced in retrospect. These problems are best solved by probe-sense-respond approaches – the kind of tight-iteration and frequent feedback approaches proposed by Agile.

Obviously, software development is a complex problem, as the cause and effect relationship cannot be fully predicted in advance, as anyone knows who has ever had to debug a program that definitely did not do what it had been predicted to do.

But I have often heard arguments against Agile, stating that certain difficult problems are solved quite well by “waterfallish” approaches, and this is seen as proof that a waterfall approach will also work for equally difficult problems in software development.

However, using the informal colloquial word “difficult” lumps various types of problems into the same category. What these problems have in common is that they appear to require a high amount of specialized skill and that many things can go wrong when they are solved – both of which is true for complex and complicated problems.

One example that I have heard is the building of a house. The argument is that a house is built based on a predictable time schedule, using a blueprint that was developed by a skilled architect. If a “difficult” task like building a house can be done in such a waterfall-like approach, then this approach should also be good enough for software.

The only houses that are “simple” (in the Cynefin meaning of the word) to build are Lego houses built unchanged from instructions in the box, as the way Lego stones fit together is near-100% predictable. But beyond that, building a house definitely requires a high amount of various types of non-trivial skills. Beyond that, I fully agree: Building a house is indeed a difficult task.

But is building a house a complex task?

Well, it depends.

There are companies which build the same house again and again. They buy a patch of land and build twenty houses there, which are identical or strongly similar to houses they have built before. Then they sell these houses, move on to the next patch of land and do the same thing. This still requires skill. This is still difficult. But these companies have perfected the process to such an extent that it has become highly predictable. I would claim that this is a complicated problem. In most cases, the planning works out quite well for these houses (let’s ignore for a moment the people who every now and then still have to wait a couple of months for houses that are completed behind schedule). And I believe this is what people refer to who say “If it works for houses, it must work for software!”

But building software is not a complicated problem like building the same or similar houses again and again. A software project is hardly ever a copy&paste from a previous project.

And let us look at another type of building projects: Completely unique one-of-a-kind buildings. As soon as a house is not just a copy&paste from a previous building project, most bets are off when it comes to time and budget planning accuracy, and the more unique the building, the larger the deviations between planned and actual time and budget. Take an extreme recent example from Germany: The Elbphilharmonie. In 2007, it was estimated that construction would take three years and cost €241 million. In the end, it took nine years to build it for a budget of €789 million. Obviously, the usual approach with detailed pre-planning did not work out. Obviously, the builders were dealing with a complex problem here.

So the argument that what works for houses also works for software is meaningless, because the only cases where a waterfallish approach really works are complicated building projects. In complex building projects with many unpredictable aspects, it clearly does not work. And software projects are complex projects.

I believe this is a common pattern in discussions about Agile. Whenever a skeptic compares software development to other types of problems, we should first consider whether the other type of problem is really a complex problem or rather a complicated problem. Just lumping all difficult problems together and proposing the same kind of approach for them does not work.