Software Development Lifecycle (SDLC) - Defect and Test Case Measurement

Test Case to Requirements Traceability - Prior to developing your test cases, you must ensure that you have an adequate number of test cases for each requirement (functional specifications).   This ensures that you have adequate testing for each customer requirement.  How do you do this? 

Make a list of every test case you have defined, and link each test case back to the requirement.  Then meet with your project team (project manager, developers and testers) to review the test cases.  This meeting will spawn new test cases you may not have thought of and will allow everyone visibility into your test process.

Basic Approach - If you do not have a software development lifecycle tool, a low-cost and simple approach to this is to create a spreadsheet that contains a list of your functional specifications (requirements) and lists each test case that has been created for each requirement.   Example: TestCase_Traceability.xls

Advanced Approach - A better approach is to utilize an SDLC tool that allows tracking of functional specifications (customer requirements), and allows creating of test cases that are linked to the functional specification.  The software should provide a traceability report that shows how the test cases are linked to each requirement.  Below are some example reports that come bundled with Software Planner, but any good SDLC tool should provide these types of reports.
Example: Software Planner_TestCase_Traceability Report.pdf     Tracking Requirements    Tracking Test Cases
 

Test Case Trending - As the testing cycle begins, each test case should be executed and marked as Passed or Failed.   As your testing cycle progresses, you should have a clear indication as to whether your software quality is increasing or decreasing.  How do you do this?

You can determine this by the number of test cases that are moving from failed to passed status over time.  Prior to testing commencing, you should have all your test cases defined and awaiting run.  Once testing begins, your team will execute each test case and mark them passed or failed.  In the early stages of the testing cycle, you may have a large number of failed test cases, defects will be generated and resolved by the development team.  As time progresses, you should re-test all the failed test cases until they all have passed (or an acceptable number have passed based on your production migration procedures).

Basic Approach - If you do not have an SDLC tool, a low-cost and simple approach to this is to create a spreadsheet that contains a daily journal of the number of test cases each day, and how many have passed and failed, with this information graphed so that it is easy to spot trends.   Example: TestCase_Trending.xls

Advanced Approach - A better approach is to utilize an SDLC tool that automatically creates trending reports.  If the software can send you a daily or weekly email with metrics, all the better.  Below are some example reports that come bundled with Software Planner, but any good SDLC tool should provide these types of reports.
Example: Software Planner_TestCase_Trending Report.pdf     Software Planner Daily Summary Report (via Email)
 

Defect Trending - As the testing cycle proceeds, defects will be found.  It is important to track the number of defects over time, so that you can determine how buggy a specific release is, and if the release is improving over time.  How do you do this?

You can determine this by measuring the number of defects and the number that are being resolved and closed over time.  As your testing cycle proceeds, the number of outstanding defects (active, open, etc.) should decrease, and the number of resolved and closed defects should increase.   It is wise to have a pre-set policy that dictates when a release is production-ready.  For example, you may have a policy that indicates that once you have no severity 1 or 2 defects, you can move the software to production.  Here are some common severities you might use:
  Severity 1 - Crashes the system
  Severity 2 - Major defect with no workaround
  Severity 3 - Major defect but with workaround
  Severity 4 - Trivial defect

Basic Approach - If you do not have an SDLC tool, a low-cost and simple approach to this is to create a spreadsheet that contains a daily journal of the number of defects each day, by status, with this information graphed so that it is easy to spot trends.   Example: Defect_Trending.xls    Defects_Advanced_Trending.xls

Advanced Approach - A better approach is to utilize an SDLC tool that automatically creates trending reports.  If the software can send you a daily or weekly email with metrics, all the better.  Below are some example reports that come bundled with Software Planner, but any good SDLC tool should provide these types of reports.  For easier data entry, it would be great if your SDLC tool allows you to automatically generate defects test cases are marked Failed.  Software Planner does have this feature. 
Example: Software Planner_Defect_Trending Report.pdf     Software Planner Daily Summary Report (via Email)
 

Other Reporting - In addition to basic metrics, here are some other reports that can provide valuable information as you manage the software development lifecycle.   Below are some example reports that come bundled with Software Planner, but any good SDLC tool should provide these types of reports.

Defect Reports:
A) Defects by Assignee
B) Defects by Project
C) Defects by Status and Priority
D) Defect Detail
E) Daily Summary Report

Test Case Reports:
A) Test Cases by Assignee
B) Test Cases by Project
C) Test Cases by Status
D) Test Cases Detail