What Peer Review Metrics are Worth Collecting and Calculating?
The first article of this three-part series, adapted from my book Peer Reviews in Software, provided some basic concepts and principles about measuring aspects of your peer review program. This article recommends several base metrics to count or measure directly, as well as a number of derived metrics that you can calculate from those base metrics to see what’s really happening in your peer review program.
Base Metrics and Derived Metrics
The basic dimensions of software measurement are size, time, effort, and quality. Although you could measure dozens of peer review data items in these categories, the base metrics listed in Table 1 (below) will give you a solid start on review measurement.
(Note that the terminology in the metrics tables refers to inspections instead of the more generic peer reviews. Measurement like this is most common when performing those formal peer reviews termed inspections.)
If you’re using standard forms for recording peer review results, include spaces on the forms to capture these values. Whenever possible, use tools to count objects such as lines of code consistently, according to your organization’s conventions.
Table 1. Some Peer Review Base Metrics
As you gain experience, you might elect to subdivide some of these items. You could separate rework effort from follow-up effort; distinguish reviews of new, modified, and reused code, or separate the metrics for major and minor defects. You should only increase the measurement complexity when you have specific questions that require more detailed data. I recommend you begin by collecting all the items listed in Table 1. Not only is the effort needed to capture and store this information is relatively small, but it’s impossible to reconstruct the data if you decide later that you want it.
You can calculate several derived metrics from the data items in Table 1 that will give you insight into your peer review process. Table 2 describes several derived metrics for reviews and shows how to calculate them from the base metrics. The computations are very simple.
Table 2. Suggested Peer Review Derived Metrics
The Peer Reviews Database
You’ll need a repository in which to store your review data, along with query, reporting and charting tools to monitor averages and trends in the metrics for a series of inspections. To get started quickly, use the Excel spreadsheets available from processimpact.com. The spreadsheet is not a robust database, just a simple tool to help you begin accumulating and analyzing the data easily.
The inspection data spreadsheet contains three linked worksheets that accommodate the base metrics from Table 1 and the derived metrics from Table 2.
- The Inspection Info worksheet contains descriptive information about each inspection, the data items in the "Other" category and the major calculated metrics.
- Enter the time and effort base metrics into the Effort worksheet.
- The defect counts in various categories go into the Defects worksheet, along with the number of defects the author corrected during rework.
- A Help Info worksheet contains instructions for using the spreadsheet and inserting new rows by using a macro.
This approach stores the data for individual inspections in reverse chronological order, with the most recent entries at the top. You can modify the spreadsheet if, for example, you wish to focus only on major defects or to exploit the charting and statistical analysis features of Excel.
The units used for measuring work product size are different for code and for other documents. Therefore, you'll need separate spreadsheets for each major type of work product being inspected: source code, requirements specifications, design documents, test cases and so on. The website contains one spreadsheet for code inspections, another for inspections of other documents, and a third with some sample code inspection data so you can see how it all works.
Now that you know what metrics are worth collecting and calculating, the final article in this series will discuss how to analyze your peer review data. I’ll also show how you can use the metrics to judge the effectiveness and efficiency of your organization’s peer reviews.