The Impact of Testing Metrics
The need to measure effectiveness and performance is what gave rise to testing metrics like Number of Defects Found, Find/Fix Ratios, Number of Tests Executed, etc. Over the last decade or two, metrics have become the natural language of management in defining the effectiveness of a testing organization. The problem with this is, of course, it is focused on the mechanics rather than the actual deliverable. Measuring the number of tests executed, for example, is one metric that has pushed some teams too far over the line into automation, leaving many bugs undetected because they require a human eye.
In his slideshare, Software Testing Metrics, PM Venkatesh Babu postulates that: “You cannot improve what you cannot measure, You cannot control what you cannot measure.” I would venture that many people will feel uncomfortable with those words, even if they can’t define why. To a developer, this would say that we will only judge your performance by the lines of code you write, not by their elegance or by the “feel” of the product you are creating. For the tester, this would say that we are not counting your contributions when you ask insightful questions at a requirements review meeting or point out the cliff the team is about to fall over.
In a metric-driven world, testers have been struggling. They can find a lot of defects but what is the quality of the defects they are logging? They can execute a lot of tests but what is the quality of those tests? And how do you measure what didn’t happen – the defects and usability problems that were prevented because a tester asked questions and provided input early in the cycle? This reduction of their work to metrics is often what leaves testers on the sidelines, feeling like they don’t have a true intelligent voice in the team.
A Different Way to Look at Testing
Luckily for the testing industry, there are some new viewpoints beginning to take hold. The idea that testers have to stay on the sidelines and accept metrics as the measure of their worth is beginning to fray around the edges as the opposition gets louder and more entrenched. For me, nothing captured the state of the testing world today more succinctly than this exchange on Twitter:
How do you get that seat at the table? It’s all about expressing your worth and offering value to the team in a way that resonates with the team members.
This became the foundation for our opening talk at the @GBSTesters meetup in June 2013. We invited Keith Klain to join us for an informal evening of beer, pizza and testing chatter. We had a lively discussion about expressing your value as a tester and making sure you not only grab that seat at the table but hold on to it as well. It was actually a great demonstration of what we as testers need to do every day with our teams – identify the issue, discuss the options, and contribute to the resolution.
Some of the ideas that came out of the discussion may help other testers out there provide more to their teams than X number of defects logged.
Know Your Stakeholders
The role of the tester in any team and any industry is to provide information to the larger team in an effort to identify existing or potential issues. But people weigh information based on their own context and values, testers included. The toughest part of the job is to remove personal bias and approach “quality” from the perspective of the project’s needs, not your own opinions. The best way to understand what matters to your project is to identify the stakeholders and what matters to them.
One way to do this is to create a “stakeholder map” that lists the people who have a vested interest in your work and in the project deliverables. Along with their name and role, spend the time to understand what matters most to them – are they concerned business potential of this feature? Time to market? Extensibility? Stability under heavy load? Usability (that’s a big bucket – what aspects of usability?)? Don’t just keep this in your head – document it and share it with those stakeholders to make sure you have it right. This is your foundation.
Express the Impact
Now that you know who you are communicating with and what they care about, you can express your findings in a way that is meaningful to them. Identifying the impact to the things that matter most to your stakeholders lets them make better decisions about mitigating issues. Remember that they are actually the ones holding the stake J – the better the information you provide and the more context you give them, the better they can be at minimizing risk. Don’t just report the issue – report the issue along with the impact it will have on the things that matter to your stakeholders.
Understand the Bigger Picture
Everything has a context. Every issue you find or predict fits into the larger picture somehow. One mistake testers often make is not taking the time to understand the larger picture. It’s very easy to get buried in the analysis of one feature in one module of one application in one project. While you’re tangling yourself in log analysis (which is valuable – don’t get me wrong), don’t get lost. Keep a picture in your head of the “You Are Here” map so you remember why you are analyzing the logs and how that fits into the bigger picture. Knowing how your daily work fits into the larger context allows you to communicate effectively and with the right amount of urgency (or lack thereof).
Let Go of the Decision
This part can be very difficult. We’ve all been in that emotional turmoil of having found an issue we believe is important but not finding anyone else who seems to care as much as we do. This can lead to a lot of fist pounding and hair pulling… and depending on how you do this, that seat at the table you wanted? You could lose it. You have managed to identify issues that matter to the stakeholders, expressed them in a way that the stakeholders can understand, fit the issue into its context within the bigger picture… good for you. That’s where you add true value to the project – now you need to let the decision makers make decisions. You’ve given them what they need to do that and you need to let go of the outcome. It’s a matter of understanding and internalizing what each member of the team brings to the table and letting them have the freedom to do their part, just as they gave you the freedom to do yours. Once you have communicated the issue and provided everything the decision-makers need to make their decision, you have done your job and done it well.
The reality is that much of what a user sees and experiences can be credited to the testers. Sure, they find the defects that keep the user from tripping over real bugs, but most testers also provide ongoing feedback that improves the overall application design… and that is impossible to measure. The testers that are most valued by their organizations are ones that ask the right questions and provide input that extends beyond defects, and do it in a way that lets the stakeholders understand the impact.