Agile and Scrum Metrics

If you’re doing agile or Scrum (primarily), you’re probably using some metrics to see how your teams are doing. Are they performing? Are they meeting their commitments? Are they on track for their milestones/releases? Are they improving? Typically, in a Scrum environment, we look at two main metrics:

• Burn-down – shows the number of hours or tasks remaining to be completed by the team
• Burn-up – shows the number of points or stories that have been accepted

I’ve seen some teams that don’t accept any work until the Sprint Demo/Review. Unfortunately, that prevents the burn-up chart from being an effective measure of team progress. Rather, I recommend strongly that teams work to complete and accept work throughout the course of the sprint, as early as possible, so that we can measure the team’s progress toward meeting our commitment. The challenge with just using burn-down and burn-up, though, is that they don’t paint a complete picture of how a team is doing. Beyond trying to improve overall velocity – the amount of work being done by the team – what would you focus on improving? How would you measure it? This is where some other metrics come in very handy. In this post, I’ll be talking about one of those metrics – cumulative flow diagrams.

Metrics: Cumulative Flow Diagrams

My favorite metric has to be the cumulative flow diagram or CFD. I think this chart tends to show a lot of information in a very compact form that allows me to see, at a glance, what kinds of things the team may want to focus on improving.

On the left is an example of a “bad” CFD. Why is this bad (or at least “not good”)? First, the team failed to meet their sprint commitments (their accepted points were a small percentage of their committed points) and spent a lot of time working on a lot of things they never finished (see my Stop Starting and Start Finishing article for information on this). They changed the scope of work during the sprint. They didn’t get acceptance from their Product Owner until halfway through the sprint despite having a story completed on day 3. The list goes on and on. With just this chart, I have at least five or six easily coachable opportunities with this team. And that gives the team more “meat” to focus on improvements over the coming sprints. I certainly wouldn’t recommend tackling them all at once. My first recommendation would be to reduce their work in progress (or WIP) so they can actually complete and deliver some value.

On the right is an example of a much better CFD from a Scrum team. Why is this one better? Because the team is actually getting work accepted during the sprint and they are delivering a lot of value. They’re keeping their WIP down to just a couple of stories and they’re working those through to completion before starting to work on more things. They didn’t change the scope of work during the sprint. And on and on and on. Is is a great CFD? No, but it’s definitely better than the one above. This CFD is certainly more indicative of a team that’s into its “norming” phase. They’re getting good at making and better about delivering on commitments, they’re partnering with their Product Owner to get work accepted early, and they’re delivering value at the end of the sprint.

Elements of a “Good” Cumulative Flow Diagram

So what are the characteristics of a “good” Cumulative Flow Diagram? Mostly, they’re the elements I listed above. First, is the team committing to work and then maintaining that commitment (not changing the work in sprint)? This would be evident in the CFD as the top line of the chart being consistent across the entire sprint.

Second, is the team completing work early and steadily? When you look at the chart, you should see the “completed” amount steadily rising and, associated with that, the “accepted” work should also steadily rise. There will likely be a delay between “completed” and “accepted” – that’s normal – but you want to keep them close together. If it’s taking more than a day or two to accept work, start asking “why?”.

Third, the amount of work in progress should be appropriate based on your team size. A good guideline for that is N – 1 where N is the number of delivery team members (minus the ScrumMaster and Product Owner). On a team of 7 people, your WIP limit should be around 6. This ensures that everyone is working on no more than one item and that they focus is on completing work rather than trying to “get everything done”, a very waterfall-ish mentality.

Fourth, your team should be shooting to get all of their work completed and accepted by the end of the sprint. We typically don’t give credit for work that’s accepted after the sprint ends (in terms of calculating velocity for future sprints). In our diagram above, we would like to see the last day show all work was “accepted”.

While there’s more to the CFD than this, these areas should give you some good starting coaching points with your teams.

Not the Whole Story

Obviously, the CFD isn’t the whole story. A team can have a great burn-up/burn-down chart and a great CFD but still be failing in a number of areas. Also – teams can try to game the chart, but I’d rather have that than ignoring the data on it. What I like about the CFD is that it’s the next layer down for a team’s performance. Like an onion, the burn-up/burn-down charts are the top layer – the CFD is the layer below that. It gives us some more information about how the team is performing during the sprint. It’s more trend data and gives us more information about how the team is tackling their work. Are they accepting work soon after they complete it or are they waiting? Are they trying to do too much at once (trying to “eat the elephant in one bite”) or are they being disciplined in tackling small chunks (“eating the elephant one bit at a time”)? Are they overcommitting? Under-committing?

Several other metrics can help us dig even deeper into the metrics onion and I’ll cover those in future posts. For now, though, check out the Cumulative Flow Diagram and see how your team is doing. What you find might surprise you.