Agile Testing Challenges - Finding Defects Early
Test and Monitor | Posted April 16, 2012

This is the fourth installment of blogs regarding the Top 5 Common Problems for Agile Testing Teams. You can view the prior blogs or download a more detailed white paper, here:

1. Agile Testing Challenges: Broken Builds

2. Agile Testing Challenges: Inadequate Test Coverage

3. Five Challenges for Agile Testing Teams: Solutions to Improve Agile Testing Results (white paper)


Agile development is a faster, more efficient and cost-effective method of delivering high-quality software. However, agile presents testing challenges beyond those of waterfall development. That’s because agile requirements are more lightweight, and agile builds happen more frequently to sustain rapid sprints. Agile testing requires a flexible and streamlined approach that complements the speed of agile.

The Challenge - Finding Defects Early

You know that defects found late in the development cycle require more time and money to fix. And defects not found until production are an even bigger problem. A primary goal of development and testing teams is to identify defects as early as possible, reducing the time and cost of rework. There are two ways to accomplish this: Implement peer reviews, and use static analysis tools to scan code to identify defects as early as possible.

Peer Code Review

We recommend doing peer code review on user stories, manual and automated tests. When reviewing user stories, you need to ensure every story is clear, supports the requirement and identifies constraints and validations the programmer and testers need to know.

When reviewing manual tests, ensure that you have enough manual tests that offer comprehensive coverage for each user story. By performing team reviews of your manual tests, it gives the team an opportunity to ensure nothing critical is missing. Programmers often suggest additional tests the tester has not considered, which prevent defects before the code goes into production.

These reviews can be done via normal over-the-shoulder reviews or you can use a tool like SmartBear's QAComplete to perform the review on-line and communicate progress and changes during the on-line review.

Reviewing Automated Tests

While developing or updating automated tests, it’s always a good idea to have another set of eyes take a look. Code reviews of scripted automated tests identify logic issues, missing test scenarios, and invalid tests. It’s critical that you check all automated tests into a source control system so you can rollback to prior versions if necessary.

There are several techniques you can apply to automated code reviews. “Over-the-shoulder” reviews entail sitting with the automated test designer and reviewing the automated tests together. As you do this, you can recommend changes until you’re comfortable with the level of testing provided by your automation engineer.

Remote teams cannot easily perform over-the-shoulder reviews. Although you could set up an online meeting and collaborate by sharing computer screens, it’s a much bigger challenge to manage time zone differences and find a convenient time to conduct a review. A convenient alternative to over-the-shoulder reviews is peer code reviews. Peer code review tools streamline the entire process by enabling an online workflow.

Here’s an example:

Your automation engineer checks the automation scripts into a source control system. Upon check-in, he uses the peer review tool to identify automation scripts that need code review. Using the tool, he selects a specific reviewer. The reviewer views the script in a window and makes notes. The reviewer can write notes on specific code lines to identify which lines of the automation script are being questioned. The tool also enables the reviewer to create a defect for the automation engineer to resolve when changing scripts. For the broadest integrations and most flexible workflow, consider CodeCollaborator, the most Agile peer code review on the market.

What Do You Gain from Peer Review?

The lack of formal code reviews can adversely affect the overall testing effort, so you should strongly advocate for them with the agile development team. When developers perform regular code reviews they identify logic errors, missing error routines, and coding errors (such as overflow conditions and memory leaks), and they dramatically reduce the number of defects. While code reviews can be done over-the-shoulder, it’s much more effective and efficient to use the right tool.

How Can Static Analysis Help?

The testing team also should strongly advocate the use of static analysis tools. Static analysis tools automatically scan source code and identify endless loops, missing error handling, poor implementation of coding standards, and other common defects. By running static code analysis on every build, developers can prevent defects that might not be discovered until production.

Although many static analysis tools can supplement the quality process by identifying bugs, they really only tell you that you have a bug. They can’t tell you how dangerous the bug is or the damage it could cause. Code review provides insight into the impact of the bug—is it a showstopper or is it trivial—so developers can prioritize the fix. This is the main reason that SmartBear recommends that you use both code review and static analysis tools.

What Are the Most Important Peer Code Review Metrics?

Focus your metrics around peer code review progress and the number of defects discovered from peer reviews. Consider these:

  1. Lines of Code Inspected: Quantifies the amount of code that’s been inspected.
  2. Requirement (User Story) Review Coverage: Identifies which user stories have been reviewed.
  3. Manual Test Review Coverage: Reports which manual tests have been reviewed.
  4. Automated Test Review Coverage: Shows which automated tests have been reviewed.
  5. Code Review Coverage: Identifies which source code has been reviewed and which source check-ins remain to be reviewed.
  6. Static Analysis Issues Found: Identifies issues the static analysis scans found.
  7. Defects Discovered by Peer Review: Reports the number of defects discovered by peer reviews. You may categorize them by type of review (user story review, manual test review, etc.).


What Can You Do Each Day to Ensure Your Testing Team Is Working Optimally?

Each day, your testing team should:

  1. Perform Peer Reviews of user stories, manual tests, automated tests and source code. Log defects found during the review so you can analyze the result of using this strategy.
  2. Automatically Run Static Analysis to detect coding issues. Review the identified issues, configure the tolerance of your static review to ignore false positives, and log any true defects.
  3. Review Defect Metrics related to peer reviews to determine how well this strategy helps you reduce defects early in the coding phase.


Be sure to check out my next installment, Web Services Testing Issues.




By submitting this form, you agree to our
Terms of Use and Privacy Policy

Thanks for Subscribing

Keep an eye on your inbox for more great content.

Continue Reading

Add a little SmartBear to your life

Stay on top of your Software game with the latest developer tips, best practices and news, delivered straight to your inbox