3 Approaches to Reporting Test Status to The Business
You are walking into work and happen to catch the same elevator as the CTO going up to the office. You have 12 floors to go and it feels like time is moving in reverse. Your CTO looks over at you and asks how the latest release is coming along.
Your brain starts racing faster than it should on a Monday morning. Do you tell her about the tests that have already been performed and how many are left? Do you talk about coverage models and areas of risk? Do you talk about how the build has been broken for the past two days and how the testing group has been stalled because of that?
Your testing elevator pitch will give a clear view of the project in just a few minutes. Ideally without any eyes glazing over from rapid fire technical jargon.
Let’s take a closer look at test reporting and your elevator pitch.
Thinking About Coverage
To get better reporting, we have to do better testing. Or at least get a better picture of the testing you are doing now. Even if we do use a scripted approach, there is so much happening that is off of the map.
Collecting some of that information can shift the conversation. There are a couple of ways to attack this problem.
Old-School Approach: Note Taking
When I work, I almost always have a pen and notepad next to me, or a mind map open, or both.
I was working on a feature refactor recently for a webpage designed around configuring alerts for a lab sampling tool. The page had more than 20 fields, some were in collapsible panes. If I tried to plan the work with test cases, I'd never get started with the actual testing. I attacked the plan by starting in on the test work right away and keeping an inventory of what I was working on.
There are a couple of ways to chart the territory that jumped out at me. We could make a list of each field on the page along with a few test ideas to get started, we could categorize by the type of alert we were configuring (email or report), or what the alert would be alerting on (samples or orders). If we pick only one category, we get a stilted view of where testing is going. Not because the others are being avoided but because, like with test case, we are ignoring everything that is off the map. I ended up with notes for each of those categories plus a few others. As I got test ideas and performed them, I'd make a note about what I did or what I found.
Note taking like this completely changes status reporting. Instead of saying I had performed 20 of the 30 test cases that were there for me, I can reference the important parts of the feature; what areas I have tested and feel OK about, the areas I have tested and don't feel so good about, and the areas that haven't been touched at all.
More Technical Approach: Heat Maps
Heat maps are a little more technical. These usually have little bits of code that get installed on the testing server that can give some insights on which parts of the program are being called. Usually these tools look at the class or method level.
So, let’s say you are using Facebook. You scroll through your feed looking for something interesting and come across a post from your friend that you want to comment on. You make a comment and then click your friend's profile. How much coverage do you have? Just based on that description it is hard to say. Heat mapping tools will give you a chart with the classes and methods that you covered highlighted. So for this case you might see Feed, Profile, and Thread highlighted in the heatmap.
This sounds pretty interesting from a coverage perspective. Now you can see exactly which parts of the code base have been called and sometimes how much. But, it doesn't tell you much about the testing that happened. I like to flip this idea on its side and think of heat maps as a way to highlight what has not been tested. A highlighted class can tie in well with your testing notes to create a story about what you have done so far. A class that has no highlight is like a big sign that says "TEST ME".
The Seamless Approach: Test Management Tool
Selecting a test management tool can alleviate much of the manual work that comes with managing test cases and reporting on test coverage. A test management tool should integrate with your existing processes and provide the end-to-end traceability you need to ensure test coverage and make sure everyone involved in the testing process — including developers, manual testers, and QA engineers — are on the same page.
A robust test management tool provide real-time insights into your testing state through built-in traceability across requirements, tests, and defects. You can setup dashboards to track which tests have run and past, are still waiting to run, or which tests have failed. You’re also able to run report reports which can easily be exported in multiple file formats (pdf, xls, rtf) and shared with your team.
Now you have a way to talk about test coverage that is much deeper than test cases, but how do you make that work for you?
Talking To The Biz
Just like in testing, understanding who the customer is will shade the information we hunt down and deliver. The most relevant question is can we release right now, and if not now then when?
If someone at the director level asked me how testing was going, I'd say something like "I'm about halfway through it right now, there are a few things I've found that would be a big problem if we released now. One of those issues is slowing testing progress." That would take 30 seconds to say, but there are bits of information that are useful to the business. Knowing that there are important issues will help that manager understand that now isn't the time to release. Managers want information that they can either take action on or use to give people even higher up in the organization an idea on how the project is going.
If your product is time sensitive and it is possible to get information on how much a delay would cost, you'll get bonus points. Being able to say that a bug in checkout when the customer uses coupons on Black Friday will cost the company $500,000 puts things in perspective.
Reporting test status can be a complicated job. There are bits of information from all over the project that has to be put together to tell an accurate story. Combine that with the fact that when we report outside of the technical group, they want a testing status that is relevant to the mission of the business and not just the development team. A deeper understanding of the testing you do is a good place to start.
Test management in an agile world
By improving efficiency and reducing waste in the testing process, test management helps to better prioritize while simultaneously reducing the time teams spend on problems after the software is delivered.
Learn more abou implementing a test management strategy in our eBook: Test Management in an Agile World: Implementing a Robust Test Management Strategy in Excel and Beyond.
Get your copy.