There Ain’t No ROI in Software Testing
Using a return on investment calculation for testing (or test automation) is an error. It’s misuse. It’s kinda like my misuse of English grammar in the title up there … But I doubt you had any trouble understanding my meaning, and you may already understand my “joke.” Regardless, it’s still wrong.
Let's clear something up first: There’s no ROI in testing unless you are selling testing services like outsourced testing. Only then does testing generate tangible, traceable, 100% correlated revenue. And therein lies a rub. ROI calculations are based on revenue-generating activities and, by definition, are a quantitative analysis method. We should use quantitative algorithms to assess tangible things like number of items sold, or amount of profit, and qualitative measures to evaluate intangible things, like quality (or the “total” value of testing).
So am I making a big deal out of nothing? I don’t think so.
Let’s use a common example: coupons and discounts. Marketers want you believe that you’ll “save” money using coupons and buying when promotional discounts are in effect. In reality, you’ll save money by not spending any. When you make a discounted purchase, you are not “saving” money, you are simply spending less. Splitting hairs, you say? Look at the bank account at the end of a shopping trip and you tell me. Did you spend or did you save?
Testing costs you, just like a shopping trip, and you might justify your actions by listing a set of benefits obtained or needs that were met, but not in money RETURNED. (A nitpicker’s note: a rebate is just a coupon-- money returned--after the fact) The fundamental purpose of an ROI calculation is to determine how much revenue or profit will likely be generated as a result of particular activities and their associated investments. It helps people make business decisions like whether this project is worth pursuing, or determining if an effort is going to break even or lose the company money.
So back to the big deal. After a big shopping trip, do you give status reports and brag about how much you spent, or do you put a spotlight on how much you saved? Why? Simply put, either choice is manipulative. You are manipulating how the information will be perceived, and possibly used. Finance is all about manipulating numbers to tell a story. Which story? Whichever one you want to tell. This is the number one reason I changed my major in college from Finance to Management of Information Systems halfway through the pursuit of my Bachelor’s degree. I wanted to major in fact-finding and reporting, not storytelling.
Information about testing is predominantly used by stakeholders to make decisions. So I think it’s important to consider what decisions about testing could be influenced by twisted information. The most common misuse of ROI in testing is with regard to the marketing of testing tools and the use of test automation in general. Why do they do it? To get you to buy tools and invest in automation. What’s really happening? Your company is spending money on tools and investing in automation. What’s the return (revenue potential)? There is none. It costs. Sunk costs in accounting terms. So am I saying don’t buy test tools or automate? NO! I’m suggesting we stop lying about the costs of testing and help stakeholders and teams figure out how to balance the investment in testing with the benefits it can provide. It’s called a cost/benefit (or cost-based) analysis. And while quantitative in nature, it’s accurate.
Today there is so much pressure in software development projects to reduce costs and timeframes, and a common target is testing. Testing is expensive and time consuming, and often the benefits aren’t understood, aren’t seen, don’t have dollars associated with them, or they truly aren’t there. When this happens, executives don’t want to fix testing, they want it to impact the schedule and the bottom line less. So we get mandates to do less testing, leaving the scope and quality goals intact. There’s no such thing as doing more (or even the same) amount of testing with less. You do less with less. But when you can be more efficient by doing the key testing first, reducing redundancy, or automating effectively, the benefit received from doing that testing may cost less in terms of time and expenditure. There’s the real story.
Testing costs! More testing costs more. And less testing often costs more than more testing! If you follow that logic, help me stomp out the misnomer of the ROI of testing. Then you can have a real conversation about what testing needs to deliver and what the company or project team is willing to do to get the benefits of the testing they fund.