Waterfall and Scrum

Many organizations find that they operate two different processes at the same time – Waterfall and Scrum. While Scrum can be scaled across the organization or enterprise (look to Spotify for examples of how to do this well), most groups within an organization that use Waterfall do so because over time they’ve built up processes that support it. Changing those processes is hard (and change itself is hard), so they tend to continue on doing what they’ve been doing, rather than experimenting. Can Scrum work within a Waterfall model? It can.

 Waterfall in a Nutshell

So what IS Waterfall? It’s the process you’re probably most familiar with (unless you’ve been exclusively involved in “cowboy coding”). The methodology consists of different phases – Requirements, Architecture, Design, Implementation, Validation, and Maintenance. It fits very nicely into mental pictures of how things should work: figure out what you want to do, do it, test it, and deploy it. Except it doesn’t work as advertised. And, according to the author of the paper Dr. Winston Royce, it would never work.

Wha

In fact, here’s what he wrote in his paper just below this diagram:

I believe in this concept, but the implementation described above is risky and invites failure.

So what did Dr. Royce think was a good idea? Feedback loops from downstream phases into earlier ones. In other words, keep going back and re-evaluate what you’re doing. While he doesn’t quite get to the concepts of agile, he does encourage the practitioner to do more than just go through one stage to the next and follow a rigid process.

But…?

But why did Waterfall take off? I believe it’s because it’s relatively simple. It conforms to a command and control style of management which was prevalent in the 1970s and 1980s. By the 1990s we were beginning to explore the concepts of agile more fully until we could end up with more formalized implementations in eXtreme Programming, Scrum, and others.

Command and control systems mirrored the hierarchies of corporations doing programming in the early days of micro-computing. Most corporations modeled themselves like IBM and DEC – developers reporting to a manager with a project manager to oversee and direct the work. That manager reported to another manager and so on. Managers were directed what work to do, they directed their employees, etc. The project manager role became the central command post for developing projects and reams have been written about how to drive risk from schedules, how to interpret developer estimates (rule of thumb: double them), how to minimize the “cone of uncertainty”, and, sadly, how to assign blame to scapegoats (related to “blame storming”). The problem with Waterfall, basically, is that it provides the illusion of control of something that can’t be controlled that way.

Sounds Like Waterfall Sucks

I would rarely use Waterfall for a development project these days. Rather, Waterfall tends to fall into a bag of tricks I have, and a trick with a very defined purpose. Unfortunately, a lot of corporations have built up their processes around Waterfall. In any large corporation, dollars to donuts, the finance team will require that funds be allocated on a defined project scope with projected ROI. Not that this is a bad thing^TM – we should know what we’re building and what we expect to see as a result of those efforts. It doesn’t take into account the potential for fixed-feature projects – it’s primarily useful for fixed-date projects (burn rate * # of days = cost). Changing those processes is difficult, though, and will move at the speed of bureaucracy – if at all.

Waterfall does have its uses, though, and many groups that have interlocking teams with different processes and different needs sometimes find that Waterfall can link them together effectively. I’ve seen this done a few times successfully, but it can still be fraught with danger.

Scrum and Waterfall – Water and Oil?

In general, Waterfall looks at the big picture and tries to project (and control) what will be developed when. Scrum recognizes that we don’t know everything at the start of a project and that things will change before we done – why not embrace change? So how can we make these two things work together?

The proposal I make most often to groups that have to do Waterfall but want to try Scrum is to focus on the “design” and “implementation” phases of Waterfall. Can we combine those into Scrum development? Ideally, let’s combine everything upstream of maintenance into a phase where we do “Scrum”. All that time we would normally spend on requirements analysis and writing documents? Focus on getting the backlog created and refined. Define some releases. Start your sprints and start delivering. The timebox for the project really isn’t going to change – why not start delivering something early and often?

If you find that you’re being boxed into a “implementation” phase only (and I’ve seen this), use the same strategy. Deliver the highest value things first and, if time runs out, you deliver what you’ve completed. (You have been meeting a Definition of Done and making sure each sprint is fully complete, right?) Then while you may not have completed all of the scope initially defined for the project, you have delivered – thanks to your Product Owner’s backlog refinement skills – the things that matter most. And you can begin to have the conversation about whether the team should continue on that path (and go over schedule) or whether they should release what they’ve completed to date. That’s a much better conversation than “we have 80% of 80% of the features done and will need twice the time for the remaining 20%”.

Closing Thoughts

I would love to say that in every organization I’ve been worked with that we’ve been able to abolish Waterfall, but that’s not the case. Too many groups are still too indebted to it – for a variety of reasons – and sometimes we have to work around it. While it’s ideal to have a team that is focused on delivering value on a product, it’s pragmatic to recognize that we’re not alone in this. There are other groups involved, other business processes that drive those groups, and that sometimes we have to find a way to fit in the box(es) we’re given. But if we can drive to have control over those boxes, we can find ways to make Scrum work in a Waterfall world.