Experienced quality assurance professionals know that there are numerous tradeoffs in delivering high-quality software on time and budget. The simplest way to describe these tradeoffs is via the decades-old Iron Triangle project management graphic that says three interrelated factors – scope, cost, and time – determine the quality of a project and when a project runs into difficulties, at least one must vary otherwise the quality of the work suffers.
When confronted by difficulties like software defects or bugs, traditional software project managers often will give their customer the options of Fast, Good, or Cheap, and tell them to pick any two. This is a way of deflating the expectations of customers who naturally want high-quality software with a lot of features and few bugs. Following the logic of the Iron Triangle, customers then have three choices: to reduce scope; increase resources (and therefore cost); or to change the deadline.
Compared to a traditional waterfall software project, scope on an agile project is not rigidly managed but is left flexible. The key determinant of success on an agile project is customer satisfaction. Agile software development teams embrace change and understand that requirements will evolve throughout a project, which is why agile methodologies allow requirements to be defined iteratively in the product backlog. In the Scrum methodology, the product backlog is an ordered list of requirements that the Scrum team maintains for each product. The backlog changes as business conditions change, technology evolves, or new requirements are defined. Continuous customer involvement is necessary on agile projects since the customer must prioritize the requirements and make the final decision about which ones will be addressed in each new iteration.
The Iron Triangle graphic defines project quality as a function of Scope (what must be built), Time (when it must be built by) and Cost (how many resources will be devoted to the project).
Image source: PMI
On Scrum projects, the Product Owner (PO) is the member of the agile team who serves as the customer proxy and is responsible for prioritizing items in the team's backlog such as user stories and bug fixes. The PO prioritizes items on the product backlog based on considerations such as business value, risk, dependencies, and date needed. Any proposed changes to working software – bugs as well as enhancement requests – are usually tracked and managed using bug tracking systems like Bugzilla or issue tracking systems like JIRA.
One Agile Strategy For Managing Bugs*
There are two general strategies for managing software bugs on an agile project. When a bug is detected, the first order of business is to try to determine how critical it is and what impact it will have on the functionality of the application or entire system. While different organizations have different ways of classifying bugs or defect trends, a generally accepted taxonomy has the following severity levels:
Severity 1: An error that prevents the accomplishment of an operational or mission-essential function, prevents the operator/user from performing a mission-essential function, or jeopardizes personnel safety.
Severity 2: An error that adversely affects the accomplishment of an operational or mission-essential function and for which no acceptable alternative workarounds are available.
Severity 3: An error that adversely affects the accomplishment of an operational or mission-essential function for which acceptable alternative workarounds are available.
Severity 4: An error that is an operator/user inconvenience and affects operational or mission-essential functions.
Severity 5: All other errors.
In many organizations, these severity levels are often labeled by terms like "critical", "high", "medium", "low" or "trivial/cosmetic." These severity levels are also not standardized since impacts vary by industry: a bug that causes a video game to crash has much less impact than one in flight control software or code that’s running in a medical device. That's one of many reasons for placing the severity of a bug in a separate category from its priority for fixing, as well as quantifying and managing the two separately.
Critical bugs, or showstoppers as they are often called, are so severe that they prevent you from further testing. But a critical bug that, for example, causes an application to crash may be a low priority if it happens very rarely. Priority for fixing bugs should be based on the risk potential of the bug. On fast-paced agile projects, bug fixes for low severity bugs often get low priority and are usually only scheduled when time is available.
Risk-based software testing looks at two factors – the probability of the bug occurring and the impact of the bug when it occurs. High impact/high probability bugs fixes should be scheduled first: for example, a bug in a particular module of code for an on-line shopping cart algorithm that keeps your business from processing transactions. On the other hand, a bug that introduces a very slight rounding error in that same transaction should have a lower priority.
Use a risk matrix to determine the priority of multiple bug fixes.
Image source: manageevents
A good shorthand way to determine the priority of multiple bug fixes is to multiply the two vectors (Impact and Probability) using a standard risk matrix (see figure 2 above). The product of the two is the Priority. Test management software is also useful in keeping tabs on bug activity, identifying the length of time to find and address defects, as well as reporting on test case progress and bug status by priority.
Although some Agile teams prefer to deal with bugs in a separate backlog, teams practicing Scrum generally find that it's cheaper and more efficient to handle bugs in the product backlog itself. Dealing with bugs on the product backlog can often be done effectively via "Three Amigos" meetings, which involve a product owner (or a business analyst), a developer and a QA tester who get together (either face-to-face or on-line) to review the bug, the tests used to detect it, and any other dependencies. In some cases, this means that the product owner or an analyst, as a representative of the business, will take the lead and ask a tester and a programmer to look more closely at the bug based on risks to the business. In other cases, a tester or programmer may be the person initiating the meeting and setting the agenda. The advantage of Three Amigos meetings is they provide orthogonal views on the bug under discussion: the business representative will explain the risks to the business; the programmer, the implementation details; and the tester offers opinions on how to solve the problem and what might go wrong. Three Amigos meetings on Scrum projects don't have to be limited to three people. In some cases, a tester might want advice from a security expert or a programmer may seek recommendations from a database or infrastructure person. The goal of a Three Amigos meeting should be to encourage at least three different viewpoints, as well as some kind of consensus about the importance and priority of the bug fix. Here, too, real-time test management tools can help aid collaboration among agile team members.
A Second Agile Strategy For Managing Bugs
The second general strategy for managing software bugs on agile projects is to avoid them in the first place. There is a school of thought that says that a problem caught in development is not a bug since the software is still being worked on. If your organization is releasing buggy software into production, however, this is an indication that you have problems with your development, testing or delivery processes.
Agile is all about short, flexible development cycles that respond quickly to customer demand. DevContinuous Delivery (CD) is a software development strategy that optimizes your delivery process to get high-quality software into the hands of your customers as quickly as possible. The notion of releasing a prototype, or minimum viable product (MVP), is crucial for getting early feedback. Once the MVP is released, you're then able to get feedback by tracking usage patterns, which is a way to test a product hypothesis with minimal resources right away. Every release going forward can then be measured for how well it converts into the user behaviors you want the release to achieve. The concept of a baseline MVP product that contains just enough features to solve a specific business problem will also reduce wasted engineering hours and a tendency for feature creep that often leads to buggy software.
To do CD effectively, your development groups should also be doing Continuous Integration, which is a process where developers check code into a shared repository several times a day. Each check-in is then verified by an automated build, allowing teams to detect errors and conflicts as soon as possible. If your company's developers are integrating code into a shared repository several times a day, testing needs to be done continuously as well. This means running unit tests, component tests (unit tests that touch the filesystem or database), and a variety of acceptance and integration tests on every check-in.
If your developers are practicing test-driven development (TDD), they'll have already written unit tests to verify each unit of code does what it's supposed to do. Automated unit tests are also the first tests that should be run as part of an automated build because they can provide the developer with immediate feedback regarding potential issues in the latest release.
Because deployment is carried out automatically as software moves through the CD pipeline, there is a risk that the newly released software that may not be production-ready. To achieve zero downtime, agile teams practicing CD will usually take advantage of a couple of different deployment strategies: “canary releasing” or “blue-green deployment.”
The "canary releasing" strategy involves releasing the next version of your software into production, but only exposing it to a small percentage of your user base. After it passes a number of environmental tests, you then release it to more servers in your infrastructure and route more users to it.
The blue-green deployment strategy involves setting up the latest version of your application on an identical clone of your production application stack, and then switching traffic from the current production stack to the new one as soon as the application passes all the manual and automated tests in your delivery pipeline. This approach works especially well if you're using cloud resources and virtual infrastructure to setup and manage your CD pipeline.
Beyond the Iron Triangle
Agile projects are more flexible than "iron triangle" projects
Agile projects are more flexible than projects that use the "iron triangle" game plan described earlier, which locks teams into delivering a piece of software within budget and on time that meets rigid scope specifications. Agile project management accommodates change as the need arises, and scope, schedule and/or cost vary as required. This flexibility comes because agile teams engage with customers and other stakeholders throughout the project--to do things like prioritizing bug fixes and enhancements in the team's backlog-- and not just at the end of the project.
On agile projects, quality comes through collaboration. Effective collaboration is vital for prioritizing bugs in the risk-based software testing approach described above, as well as throughout the entire Continuous Delivery process. An agile team that uses a test management tool that allows them to work collaboratively in real-time will also be able to recognize defects in their products and mitigate them more quickly. This shortens development and testing cycles, improves team efficiency, and reduces the time it takes the team to bring high-quality software to market.
Related Articles:
*(Watts S. Humphrey "Managing the Software Process" (c) 1989, Addison-Wesley, p. 218)