Agile metrics – the good, the bad and the ugly

Agile teams often look at and discuss metrics. Some of these metrics are interesting and important, some of them are of questionable value, and some should only be used with caution. This article will attempt to clear this up.

The use and misuse of metrics

Before going into the various metrics that agile teams use, it is first important to understand the context and purpose of using metrics. Agile metrics should be used by a team to perform introspection on their own performance, or to make calculations to feed into their own release planning. If people outside the team, such as middle managers, want to start inspecting metrics, that is usually a warning sign. Metrics should not be used to compare one team against another team, because the teams are operating in different contexts and building different things. Be wary of anyone who wants to use metrics, especially velocity, to find the “underperforming teams”. If they do, I give you permission to ask them if they have metrics on who the underperforming middle managers are!

Burnup / burndown

burnup chart agile metricsThis is a simple and useful one, and to be honest it is more a representation than an actual metric. A burnup chart shows the rate at which the team is delivering stories compared to how much work there is left to do to complete the release. The burndown chart is basically the same but inverted; it shows how many points remaining are to be delivered in the release as the team completes work. The scrum master and/or project manager and/or product owner will probably want to look at these at some point; that’s fine. My only caveat is that in order for them to be useful, the entire backlog sometimes needs to be estimated, something that I don’t think is a good idea. I suggest instead that you just use average points for all of the stories you haven’t estimated yet. So for example, you’ve estimated 20 stories, at an average of 4 points per story. You have 50 more stories in the backlog that you haven’t estimated yet (that’s actually fine, I recommend teams to estimate as they go). You can average them out at 4 points each which means your backlog is roughly 200 points.

Velocity

This is the  one that you need to be careful about, since it is frequently used by managers to measure “performance” or “efficiency”. Velocity is simply the number of points the team has delivered on average per sprint. Some important caveats:

  • You need at least two or three sprints under your belt before you can establish a meaningful average
  • You should probably use a rolling average of the last four or five sprints (data becomes less meaningful as it becomes older)
  • It should be used for release planning
  • It can maybe be used for sprint planning (but not necessarily)
  • It should not be used for anything else, especially not comparing performances of teams.

Story cycle time

Story cycle time agile metricscycle time is a good metric and one that is not used enough. It is the average time it takes for a user story to go from Ready for Dev (or whatever your equivalent Kanban state is) to Complete (or whatever your equivalent Kanban state is). Different organisations have different definitions of “ready” for development or production, and that’s ok, but try and be consistent within the same organisation. You want your story cycle time to be low, ideally around half of your sprint length. Remember, a story should be something that can be completed in a sprint. Some will be a bit bigger, some will be smaller, so if your average is half your sprint length, you should be able to complete most or all of your stories in a sprint. And that’s a good thing.

Story lead time

Story lead time is like cycle time, but it is the time for a story from when it is first created to when it is completed. It encompasses but surpasses cycle time. So story lead time must be equal to or greater than cycle time, always. Cycle time is generally more useful, but lead time (expressed in relation to cycle time) tells a subtle and important story. I’ll write a separate article on these two and their differences soon.

Story count

This is a very simple metric but can be quite useful, especially with teams that are using Kanban or Scrum-ban. It is simply the average number of stories delivered per sprint. In case you’re thinking “but what’s the point of that if stories can be of different size?”, then you have answered your own question with a very profound answer: what if stories were all the same size? If you wrote your stories in such a way that they were all the same size, then your velocity would be more simply expressed as your story count, and you could do away with story level estimation altogether. Think on that one for a while!

First time pass rate

This is the percentage of test cases that pass system (ST) or System Integration Testing (SIT) the first time around. As your teams reach a decent level of agile maturity, these should approach 95 or 100% quickly. If they don’t, you have some problems.

 

Leave a Comment:

3 comments
Add Your Reply