Should code review metrics be LOCs or SLOCs?

  May 07, 2009

You normally see code review metrics in terms of LOC (Lines Of Code).  You use for stuff like "How fast do we review code?" (LOC/hour) or "Which files are the riskiest?" (high defect density == defects/LOC).

I'm frequently asked why we don't use one of the "better" measurements of "amount of code."  This can mean many things:

  1. Don't count lines that are blank or contain only white space.

  2. Don't count lines that contain only code-comments.

  3. Don't count lines that contain only a trivial code elements (e.g. just white space and a close-curly-brace).

  4. Count number of "statements."

Depending on who you ask, there are different names for these ideas.  Often someone will pick one of them and call it "source lines of code" or SLOC.  So let's call SLOC "whichever of the above you like."

The question is: Is SLOC a better metric than LOC when talking about code review?

The usual argument for SLOC is that is more closely represents the "amount of code" or "size of work product."  Indeed, for static analysis metrics like method-length or file-size/package-size, SLOC is almost certainly a better measurement than LOC.

For code review, though, I think you're better off with LOC.

First of all, you can't throw away the comments.  A major benefit of code review is ensuring that the next developer who has to maintain or use this code will be able to understand what's going on, and for this code-comments are just as important as the code.  Even just answering the question "Does this code work" depends on the reviewer reading the documentation to see if coded behavior and unit tests match the prose.

Second, eye-tracking studies done during code reviews show that the physical size of a method directly affects how long it takes to code review (reference the code review study (PDF) from my book, page 37).  The physical size (including white space and trivial code) correlates better with "time to review" (and by implication, "effort to review") than does any definition of SLOC.

Third, as an implementation detail SLOC is language-specific, which means you have to use tools that support that in all your languages.  Depending on how complex your development is, that might be trivial or it might be near-impossible.  At least LOC is a clear-cut definition that works across all text files.  Although of course LOC of assembly code is not comparable to LOC of Python, at least the definition is clear in both cases, and as an estimate of "amount of work for code review," LOC is closer than SLOC across languages.

In the end, you have to remember that any definition of LOC or SLOC is a poor measure of work, code size, or risk, so take all these metrics with a big ol' grain of sea salt.

Indeed, just the fact that LOC/SLOC is such a rough measurement provides a final reason to prefer the simple, clear definition over the language-specific tricky definition.