In my role as an Agile coach and Scrum trainer, I get the chance to talk to a lot of Scrum Masters and teams. When I ask them about what their team is measuring, the most common answers I hear are velocity, story points, and burn-down charts. Makes sense, many Scrum Masters learn about those topics in an introductory Scrum course. Velocity may be fine when just getting started with Scrum, but armed with more knowledge and acting as a servant-leader, I want people to improve and get better business outcomes. What about business value metrics, I ask? Doesn't product quality interest the developers, as well as code health and the amount of technical debt hiding beneath the surface? Team health? Flow of work, from the time the team starts working on a Product Backlog item until your customer is using it?
Did you know that velocity and story points are not even part of Scrum or found in the Scrum Guide? And are you aware there are better metrics awaiting your exploration? In this article I discuss the proper way to use velocity if you are using it, the misuses and pitfalls, why it may be time for you to move beyond it, and suggest some metrics dimensions you should explore and experiment with.
If You Find Velocity Useful, Use It Correctly
I'll be the first one to admit that as a Scrum Master I grew up teaching velocity to lots of teams, even being the one to track it for them, rather than empowering teams to self-organize to track their own metrics. I often failed to help developers understand its true purpose. As mentioned, velocity is not part of Scrum, rather it is considered a complimentary practice. Your understanding of the definition is important - read it carefully. It is defined in the Scrum glossary as follows:
An indication of the average amount of Product Backlog turned into an Increment of product during a Sprint by a Scrum Team, tracked by the Development Team for use within the Scrum Team.
What that definition implies is that the product Increment meets the definition of "Done" and is potentially shippable by the end of every Sprint. And that's important! A Development Team uses velocity to understand their capacity during Sprint Planning to build a "Done" Increment by the end of the current Sprint, and for a Product Owner to forecast the release of a one or more Product Backlog items (PBIs). It also doesn't say it is to be used by managers or tracked by Scrum Masters!
If you are using velocity to measure PBIs that are not potentially shippable or to measure partially done work, you are misusing the metric. If you think velocity measures productivity or business value, you are mistaken. If you are a Scrum Master tracking it, you're not helping your Scrum Team in the long run.
And you don't need to measure velocity with story points or even use story points! Number of Product Backlog items (PBIs) turned into a "Done" Increment may be all you need as an input into Sprint Planning, especially if you are right-sizing PBIs into one or two days of effort. Why bother with pointing?
I like to say that I may have invented story points, and if I did, I'm sorry now.
- Tweet from Ron Jeffries, May 23, 2019
When I ask the team how story points will be used, often the response is 'to get credit'. That's not a valid reason. Another common debate amongst Scrum Teams is regarding whether or not to 'point' spikes. I usually advise against that because spikes do not get turned into a "Done" Increment within the Sprint. As a reminder, spikes come from the XP community, and one way a Scrum Team may use them is for Backlog Refinement activities to gather more information on a Product Backlog item. Spikes are typically time-boxed to a set amount of hours. So if you time-box the work to a specific set of hours, isn't pointing the spike needless estimation since you already have an estimate?
The Danger of Relying On One Metric
If decisions are being made based on one dimension like velocity, red flags should go up. Let's say for the past quarter, a Development Team's velocity has steadily improved 10% on average for each two-week Sprint. On the surface that seems like a nice steady trend over 6 Sprints. Without looking at other metrics, we probably don't have enough empirical evidence to decide if it is good or not.
One metric alone may give you a false sense of security. Let's take this a step further. What if quality has trended in the wrong direction, with more defects escaping into production, which are discovered by your customers and your brand is eroding right by the minute? What if under the product's surface, technical debt is starting to rage out of control and you don't know it? In upcoming Sprints velocity will most likely start to decline, because the codebase becomes unmaintainable, product quality starts to suffer, and developers leave because no one wants to work in a crappy codebase. And worst of all, what if your customers aren't using your product or find little value in the features you just released over the quarter? How useful is velocity now?
Setting Velocity Targets Often Backfire
Scrum Masters are responsible for leading and coaching the adoption of Scrum, and should spend time with their organization to help them understand what velocity is and its proper use. If they see managers comparing one team's velocity to another, or setting targets for a Scrum Team to achieve, the Scrum Master must step in and teach the organization about more useful measures, as well as the unintended consequences which could come from setting targets. I usually start by pointing them (no pun intended) towards this useful document called the Evidence-Based Management Guide.
I once worked at a company where a yearly performance objective was set by a management team:
Every developer will increase their individual contributions towards the team's velocity by x%, measured on a quarterly basis.
Even though the managers had good intentions, as you can imagine, there wasn't much collaboration going on on within those Scrum Teams, as there was no incentive to help out a teammate. I'm also sure story point inflation was rampant, with developers inflating their estimates to provide a safety gap. It's just human nature, which is actually backed by a theory called Goodhart's Law, which states:
When a measure becomes a target, it ceases to be a good measure.
And if you're a manager whose Scrum Team is using velocity, I can't stress this enough: it is neither good or bad if velocity goes up or down. My suggestion is to not ask about the Scrum Team about it, don't compare one team's velocity with another, don't ask for it on a status report, and don't put objectives in place around it. You're just setting everyone up for failure.
Augment your Data with Healthy Conversations
In my experience metrics are not very helpful without context, and that's where conversations can be extremely valuable, generating insights, and making the data transparent. The conversations that ensue can help the team decide on the next best action to take. Transparent team-level data can serve as critical input into a Sprint Retrospective, used first and foremost by the team for continuous improvements.
Throwing numbers on a status report is ill advised, and if you're required to do so and have no other choice, try to find compromise by recommending that status be accompanied by a conversation, preferably with the entire Scrum Team present at the Sprint Review. And for you former Project Managers turned Scrum Masters, take a step back and let the Development Team and Product Owner present the data. And shift those Sprint Review conversations away from team-level metrics such as velocity. Product-level data can make for an effective Sprint Review, generating effective conversations with your stakeholders. Be open with your stakeholders, their feedback can be extremely valuable.
Here's the million dollar question: should team-level metrics be shared with managers and the organization? Put yourself in the manager's shoes, should they really be left in the dark? The answer depends on which metrics we are talking about, and how much trust has been built with leadership. Before any metrics are shared and made transparent outside of a team, Scrum Masters should work to build psychological safety and establish trust
between the team and leaders. Invite your manager to the Sprint Review. Ask your Scrum Team if there is an information radiator they might want to share in their team room, such as a release burn-down chart or a cumulative flow diagram, and make sure to educate and teach the managers about what they are viewing.
Once trust is broken between a manager and a team, the metric will eventually cease to be valuable. The best advice I will offer to managers is to use the data for learning, not judging, incorporating humble inquiry into their conversations with the team. Managers can best serve the Scrum Team to understand and remove organizational impediments which get in the team's way.
Focus on Outcomes over Outputs
Outcomes focus on real value delivered to your customer, whereas outputs such as velocity measure how much is produced. Focusing on outcomes over outputs makes a great case for why the velocity metric isn't useful outside the team. Imagine a stakeholder or leader thinking great progress has been made because the team's velocity keep getting better Sprint after Sprint? Eric Reis, in his highly recommended book Lean Startup sums it up nicely:
“What if we found ourselves building something that nobody wanted? In that case what did it matter if we did it on time and on budget?”
Outcomes may help answer the following questions:
Are we building the right product features? And building them right?
Are customers happy with our product? How often do they come back?
Are they using our product, and in which ways?
Closing
Metrics are a hot topic within the Agile community. In my experience Scrum Masters and their teams remain cautious around metrics, and rightfully so. What holds them back the most is the lack of trust and fear of misuse by organizational leaders and their management. As a Scrum Master it is your job to work with the organization to help them understand metrics and the best way to interact with the Scrum Team.
There are hundreds of metrics available to you. If you want to take metrics a step further, here are some categories I would recommend that you should explore and experiment with:
Business Value
Flow
Team Health
Quality
If you're fortunate, at some point your Scrum Team will evolve beyond those basic measurements of velocity and points, moving into their next level of maturity as trust starts to build within the organization. Perhaps this shift starts with your Product Owner deeply immersed in Google Analytics, or your development team starts to get a good handle on technical debt with the use of data such as cyclomatic complexity. Maybe your Scrum Master comes back from a cutting-edge course, and starts to teach the team about flow measurements, such as cycle time. As a sign of this taking shape, information radiators may start to appear in the team rooms, and metrics conversations happen with stakeholders during the Sprint Review.
Comments