One reason why methodologies like Waterfall have been popular is because of something I call the “Illusion of Control”. We think that, by identifying everything we can think of and tracking it down to the finest points, we can control the process and keep things from getting away from us. That by knowing everything we can know about something we can prevent bad things from happening. The problem is that it’s an illusion that we have control. This is part of the reason why 91% of Waterfall projects at large companies fail to meet scope, date, and/or budget.¹

The Waterfall Illusion

The best laid schemes o’ Mice an’ Men, / Gang aft agley

– Robert Burns, “To a Mouse, on Turning Her Up in Her Nest with the Plough”

I see in a lot of organizations that are working in Waterfall or “Cowboy Coding” mode that when things seem to be out of reach – we can’t get all of that done by this date – that sponsors and management want to be reassured by having a plan. They ask for detailed plans about what needs to be done. Can we compress this? What about spending less time here? And when the plan is created and everyone has anted up something into the kitty to “make the date”, those sponsors and managers feel happier. They saved the project! It took some cajoling and some pressure, but in the end the team came up with a plan. And the team agreed to it! All is good! Now that we know exactly what we need to do and when we’re going to do it, we’re going to be successful!

The Reality

No battle plan ever survives contact with the enemy.

–  Paraphrased from Helmuth von Moltke (the Elder)

As any project manager will tell you, though, a plan pushed through to cram functionality in by a date is doomed to failure in one of several aspects. Either something will happen that pushes the date out (and the commitments aren’t met) or something has to be dropped from the feature set (if the date is the primary goal) or they have to add so many people to the fire drill the project’s become that they blow the budget and then some (almost 53% of projects spend 189% of their original budget)². In all of these cases, we have failed to account for “life” happening here. Chaos is the enemy of well-planned projects. And chaos is everywhere. Sorry my son got hit by a baseball and is in the hospital. Sorry my car broke down and I can’t come in today. My water heater exploded and I have to replace it and clean up the mess. And yes, I’ve heard all of those reasons.

The trouble is that crammed plans are destined to disappoint because they fail to account for chaos. “Why not just add in buffers?” you ask. Ahh. That’s been tried. You’ve probably heard the project manager joke about the estimating rule for developer estimates: whatever they tell you, double it. Why? Because some things are harder than other things and take more time. Because what we thought would be easy is turning into a gigantic problem. Because Bob in operations tried upgrading a load balancer and now we can’t access our servers for two days. Whoops. Chaos. But buffers can’t fix everything and the first things that go, when you try to exert more control over a project to “bring in the date”, are the buffers.

Most of the time, processes have sprung up to try to control past failures or deviations. Finance teams seem especially apt to have adopted these – and, in many cases, understandably so (given the cost overruns noted above). But those controls and checks and stage-gates and everything else that’s done to try to fix and control a chaotic process often lead to more problems than they solve. I’ve seen projects that should have taken no more than a month take three because an approval process (which added absolutely no value) was instituted to control waste. I think that qualifies as “irony”.

Fixing the Waterfall Illusion

So what can we do to fix the illusion of control we think we have? First, like all 12-step programs, recognize that we have a problem. Tracking every niggling detail is not going to give you more with less. It won’t even give you a plan that you can execute with flawless precision. Ten gallons of water will not fit in a five-gallon bucket; stop trying to make it happen. Figure out what’s really important – the scope or the date. Once you’ve figured out what has to happen, work to be successful at that goal. Give the teams a chance to succeed on their own (see my Starting With Scrum article for some pointers). I can pretty much guarantee you that your teams want to be successful. Don’t set them up for failure just because you’re trying to “do more with less”. Rather, recognize that chaos is here and it’s here to stay.

There is a general rule that I like to quote from Bob Hartmann, the trainer from whom I took my CSM and CSPO classes, which is “Just enough, just in time”. Focus on what makes sense to work on now and then do that. This provides the team the focus on what’s most important to the business and the customer. You don’t want to be completely myopic; you want a couple of sprints’ worth of work identified, but you certainly don’t want to try to create and estimate the level of effort for an entire backlog of work. I’ve seen someone try that (hint: it’s pointless and you’re wasting your time and money; remember: chaos). Your focus, though, is on the scope or the date.

You might be wondering, though, if I focus on the date, what if the team can’t finish what I need to make a product? In this case, there are two components. First, you need to identify what’s called a Minimum Viable Product, or MVP. The MVP contains all of the functionality the Product Owner believes should be present in order to make a product that has solid value to customers. (You need to avoid the trap of asking for the MVP by a particular date.) Work with the team to find out when they can be done with the MVP and use that as the baseline for the delivery dates. To do otherwise invites the same Illusion of Control problem above. Additionally, the team should be delivering something of value every sprint. Even if they don’t complete all of the work identified, what they have completed should still have value. In fact, based upon the Product Owner’s refinement of the backlog, it should be the most important things for the customer. The items they haven’t completed yet should be lower-priority features. Is what they’ve completed sufficient to satisfy customers? Release it! If not, can you release what’s been developed now and follow on with more functionality later? Look at the options that may exist and don’t be afraid to be creative.

Trying to control the process is a very difficult habit to stop, but like all bad habits, it takes recognition that it’s happening and a desire to change things for the better. If your organization is committed to making this change and has the wherewithal to follow through, it should be a simple, gentle reminder that “this isn’t how we do things now”. With some diligence and some gut checks by everyone on your team, you should be able to get past the Waterfall “Illusion of Control” and get back to what really matters to your team, your business, and your customers.

Notes

1. Standish Group CHAOS Report: http://www.projectsmart.co.uk/docs/chaos-report.pdf
2. Ibid.