If you've ever heard, "It works on my machine," "It works as designed," or "But we need to ship Tuesday!" this article's for you. Here’s how to train users and QA to report bugs in a manner that helps the developers actually fix the problem.
Imagine a software development team using all the latest tools, the fanciest JavaScript code libraries, and the most popular, fanciest process to be found at the XPAgileScrumOneConference.
Despite that, the developers can't reproduce the reported defects, and the team does a poor job prioritizing bug fixes against ongoing development. Season with salt, bring to a boil, and wait six months.
What kind of dish do you think will be served?
Yeah. Me too. It turns out that bug reporting and triage are important, and it's something we don't talk about often enough.
Let's get to it.
Bug Reporting
Developers need to know four things to reproduce a bug:
- The operating environment. The browser, the source system, the type of mobile device, the operating system – anything that the programmer might assume wrongly about where the bug actually appears.
- The reproduction steps. Starting from a fresh environment, what does the user have to do in order to cause this problem?
- What actually happens. What goes wrong, such as a screen freeze with permanent hourglass, a pop-up error message (don’t forget to include the message), perhaps a JavaScript error in the bottom-left hand corner of the window (with the text that pops up when you click on that error message).
- What you expected to happen. This is especially important if the application has to run through a process to spit out a "correct" answer. The technical staff needs to know what that correct answer is in order to verify the bug is fixed. Personally, I recommend the "what I expected" line for everything, even down to stating that when you click the bold button, the text should turn bold. Likewise, if you've got a formal software specification, that's great; you can refer to it. In many cases I find that a human being expressing his expectations does just fine.
Now consider the finger-pointing, wasted time, and washing-of-hands that occurs with a bug report like "Cut and Paste is broken." Imagine if the person who reported the bug took the three minutes to write the defect in the format above. All that time saved means new features to customers, which means money in the bank.
If you've got problems getting users to express their problems including these four issues you can send around the link to this article. Or print it, cutting out those four elements, and tape them in convenient places. I wouldn’t object if you put it in the coffee break room.
Consistently well-written bug reports can reduce team friction, win friends, and stomp out bugs. Best of all, it's an easy habit to develop.
Reducing Friction Still Further
The four things above may be all a user needs to do to explain a bug, yet sometimes you can go further. If the error is "it just looks wrong," you might want to attach a picture. If you suspect you will be faced with shock and disbelief, consider making a video demonstrating how you can reproduce the issue. (Techsmith produces a wonderful little tool called Snagit that can create image and video captures for Windows; I own a personal copy and have for years.)
Sometimes the bug can be hard to reproduce, or at least hard to pin down exactly. If your team has a virtual environment that can be saved (using a tool like VNC or VMWare), you might be able to save the system image just before the point of failure and refer to that environment in the bug report. Alternatively, you might write something like, "Click back and forth rapidly between buttons A, B, and C; you will see an error within three minutes. &tester-name can reproduce the problem on demand if needed." This kind of minor encouragement can prevent the time that is wasted from a programmer marking the defect as UNABLE_TO_REPRODUCE or WORKS_FOR_ME and prevent hissing fits in the office hallway.
Many bugs do not lie on the “happy path” but instead only occur under specific and rare occasions. You may have heard these referred to as “ corner cases” or “boundary bugs.” I find that what a developer thinks is a corner case is a reasonable customer use. If you find yourself in that situation, I suggest documenting how the typical user might trip the bug in the defect report. This can help decision makers with setting priorities, help programmers understand the customer, and possibly help make a better end-product.
You might also have a bug field for severity. While I'm not a huge fan of terms like "minor," "major," and "medium," some defects are simply so bad that they must block a new release. In addition to the incredibly obvious problems (e.g. “login is broken”), consider a zero-regressions policy. That is, any new defects introduced into old working code must be fixed before the new code can be released. The thinking here is that these fixes are generally small, and drawing a line like this means the overall product quality tends to improve. Without some sort of stopping line, human nature, compromise, and schedule pressure invariably lead to a deterioration in product quality.
Who needs a bug tracker, anyway?
Several experts in the field recommend actively working to change your software development and testing process to make the use of the bug tracker obsolete; I'm not one of them.
It can be done. You may have a “drop everything to fix new bugs” policy, or a “zero known bugs at ship time” policy. You might have the person who finds the defect pair with a programmer, and perhaps write an automated test to cover the defect, see it fail, fix the bug, and see the test pass.
If your environment can support that kind of work, that's great – but not all companies have that luxury. You might have to deal with multiple browsers and incompatible GUI frameworks, for example, and it might make sense to defer a fix for a platform that won't be supported in six months anyway. Likewise, it’s possible that your customers prefer new code now, and are willing to live with flaky behavior, as long as you push fixes quickly or the behavior is limited to corner cases.
I can't help but notice that most of the companies with this issue are giving away free services to end users, like Facebook or Twitter, or else writing video games. Before you consider a Facebook policy, it's worth asking: Are we Twitter?
That's sort of the point: Different companies have different requirements for success, and it would be foolish for me to prescribe a solution (such as eliminating bug tracking) before examining the patient.
On the other hand, I have seen many teams succeed at decreasing their reliance on bug-tracking by communicating in-person, especially early in the process. You can accomplish a lot by having the testers talk to the developers instead of managing workflow by "tickets" and "queue." For example, the team might have a policy of logging bugs that are found externally (in production) or during regression-test so they can be tracked, but bugs found during the testing of an individual feature result in a new story-test, or new requirement for that feature, and are tracked directly with that feature. Likewise, if a developer finds a bug, he can simply fix it; no need for tracking. The key is to find a sweet spot.
Does that mean you can eventually eliminate bug tracking entirely? Perhaps. Track your bug rate for a couple of weeks, try aggressive fixing, and see what happens to team morale, product quality, and project velocity. Ask, "What would it take to get the bug tracker gone, and are we willing to go through that?" You might be surprised at the answer. (If you find that bug tracking is appropriate, or you are on the other end of the spectrum, surrounded by bug reports tracked by sticky-notes, I would be remiss not to mention our site sponsor, SmartBear Software, as a provider of such tools.)
Triage
In medical terms, Triage is the process of separating the people who need help right now, the ones who can wait, and the ones who can't be saved. In software development terms, those roughly correspond to the bugs that need to be fixed for release, the ones that can be deferred, and the ones that are uneconomical to support. No, don't look at me like that; I've known several organizations that saved a fair amount of time and money by first deferring minor issues in IE6, then eventually ending support for customers on that browser.
There are lots of ways to perform the process of triage, from "bug councils" to "war meetings" to letting one person sort the list. (Ken Schwaber, creator of Scrum, has a name for this person: The "single wringable neck.") My experience has been that most development teams can eventually get the bugs in some sorted order, by hook or by crook. The question is: How do you decide what to work on first: bugs or ongoing development? Beyond that, how far down that list do you need to go in order to ship the software?
If we ignore this problem, and "just" have two different lists, we are stuck on the same old cycle. Once more, invariably, pressure leads to compromise, and, over time, product quality suffers.
The first step is getting the bugs and new features on one list, called a release list. When programmers want new work, they "pull" from that list. When the list is done, the release is done.
Of course, there are other ways to structure the work. You might go down the list and, when the time runs out and the music stops, the team shifts from build mode to regression-test mode. On any but the smallest project, regression-testing yields more bugs, and you'll play the same game, this time trying to decide when to make the ship decision.
You'll notice the bug reporting steps were "Just The Facts Ma'am;" that was designed to reduce friction in the process. It is during triage that your opinion can come out; triage is the time to talk about product quality.
Becoming Quality Advocates
It's more than a little ironic that "QA" is only rarely capable of actually Assuring Quality. After all, programmers write the code, not the testers – and product owners and managers get to decide what is fixed. It is always possible that these decision makers agree on shipping a poor quality product. (For that matter, the specification might be for a poor product, but I digress.)
Yet we "QA" folks can be Quality Advocates. We can be in the Bug Council; we can be in the war meeting. We can say "fix it" and we can argue passionately for the fix.
One way to advocate for quality is to suggest a policy like the one I outlined above: Regressions are automatically categorized as "must fix before ship." Another policy to consider is to fix all bugs before ship; a third might be to fix new bugs as they are found, dropping all development work to do so. You might not get these policies enacted. Still, just having the discussion can have a positive effect on product quality.
You might be tempted to point out that fixing bugs (or not) is an economic decision. Shipping the night before the big trade-show or having the feature available for the Christmas rush can have great value, and it's true, and I may have to put that hat on as I advocate for the customer.
But I'm still going to advocate for the customer, and somehow, I suspect the customer would prefer that we just fix the problem.