Ingrained Biases in Software Testing That Work Against You
Test and Monitor | Posted August 23, 2013

Software testing is loaded with unchallenged ideas from the last 20 or so years, and changing its perception works against ingrained bias and prejudice.”

I recently came across this line in a blog post by Keith Klain and, as someone who loves to challenge ingrained biases and prejudices anywhere I can, it definitely caught my attention. The blog goes on to illustrate some interesting new ways of testing software, highlighting one company in particular that’s really getting it right.

But, by the end of the blog post, I was only more curious about that one line and the journalist in me started getting antsy… so I arranged an interview with the author.

Before I share the entire conversation, I wanted to address the original point that pulled me in… Just in case it piqued your curiosity, too.

What are the Major Biases Holding Back the Software Testing Industry?

According to Klain, the whole industry is underpinned by a false analogy that’s made between software testing and manufacturing. It may be false, but it’s not hard to see where it came from. Software is a product people buy, so it makes sense that it’d be treated like every other thing on the assembly line. To be fair, it may have worked perfectly fine when software was limited to being embedded in devices or shipped on CDs. But, like everything else, the Internet and wireless revolution pulled the strategy rug out from under software testing. So, if you’re actually testing Web based software today, that analogy—and the bias and protocol that result from it—is probably a major source of frustration for you.

This Is Not a New Conversation

Before we get into the details about why the bias towards seeing software testing as a manufacturing process doesn’t fly and what we can do about it, I’d like to step back a bit and acknowledge that this isn’t a new discussion. I found this blog from a few years ago calling for “manufacturing no more,” with the astute suggestion of replacing the manufacturing analogy with an attending physician for web applications:

“We just don't write or release software the way we used to. Software isn't so much built as it is grown. Software isn't shipped ... it's simply made available by, often literally, the flip of a switch. This is not your father's software. 21st century development is a seamless path from innovation to release where every phase of development, including release, is happening all the time. Users are on the inside of the firewall in that respect and feedback is constant. If a product isn't compelling we find out much earlier and it dies in the data center. I fancy these dead products serve to enrich the data center, a digital circle of life where new products are built on the bones of the ones that didn't make it.

In our father's software and Deming's model we talk about quality control and quality assurance while we play the role of inspector. In contrast, my job seems much more like that of an attending physician.”

I really like the physician analogy, even though it may be a bit of a stretch. For now, it’s a healthy one. We need to see test cases more as living organisms that must respond to changes and adapt to input. We need to keep records and track data in relationship to a dynamic system that lives in a habitat so to speak, not an unchanging object. With that in mind, let’s dive into the issue.

Why the Analogy Between Software Testing and Manufacturing Doesn’t Work

Many people who are involved in the software industry but don’t test software think you should (very easily) be able to break a test system down into predictive components. Often they are the ones looking to either make or save money by counting and quantifying all the steps it would take to sufficiently test software. So, it necessarily needs to be predictable—you can’t determine how much you could save or make if you can’t predict it. Nothing wrong with wanting to make and save money, of course.  It’s the insistence that predicting test cases makes for adequate test cases that ultimately undermines quality efforts.

This is essentially trying to create a commodity around a test case so that companies can sell testing services by promising to break down a system into hundreds or even thousands of test cases and charge for each one. Or, for a QA manager, this strategy might be understandably motivated by the need to plan a budget. But that approach robs a tester of much needed flexibility, the freedom to adapt to previous test results and their stakeholders, to change things as they get to know the software, and ultimately to produce intelligent tests that work well at identifying defects.

The other major influence supporting this old paradigm is the trend of shipping software to be tested overseas or off site. The need to control from afar and give directive guidance has deepened the manufacturing correlation rut tremendously. Again, it’s understandable. We humans usually equate control with security. Unfortunately, it’s not true in this case.

The Enormous Cost of Software Defects

A recent Cambridge University study cited that the U.S. economy looses $312 billion per year from software defects. That’s a huge number, but there are about as many ways to calculate those costs as the number itself. It’s like trying to count all the impoverished children in the world—it depends so much on how and where you look and often by who is doing the looking. So, I personally prefer to track individual stories, and particularly appreciate this slide show Bloomberg put out a few weeks ago on the most damaging software defect catastrophes over the past 50 years. The point is, either way you look at it, we have a lot of room for improvement.

What’s Being Done About It?

We all have biases, be they from language, education, culture, nation, gender, age or anything else that makes us human. In some ways, we can never be completely free from bias, but we can and need to be aware of its influence – especially when it becomes an obvious roadblock to our development. Luckily, the bias towards a manufacturing model in testing has already proven to be something we can change—with a little education, patience and persistence.

Keith Klain and others like him are taking a few different approaches to shifting this tide—one from a leadership or managerial perspective that involves a major overhaul of entire testing teams, and the other through educating the individual tester.

Only a few years ago, Klain took over Barclays’s Global Test Center. While he found that the team was extremely proficient in test management, “their misaligned priorities had the effect of continually hitting the bull’s eye on the wrong target.” Keith put a system in place “to foster testing talent and to drive out fear—abandoning worthless metrics and maturity programs, overhauling the training regime, and investing in a culture that rewards teamwork and innovation.” To hear more on how Klain implemented these radical changes, check out this YouTube video.

Obviously, the kind of overhaul Klain directed at Barclays takes a highly motivated, dedicated and persistent leader—let’s be frank, that’s a rare person. However, he and other influencers in the quality industry are working another angle as well. And this one not only is dedicated to revolutionizing software testing, they are out to put a dent in poverty as well. Through software testing training programs targeting under-served youth and adults in several major U.S. cities, Per Scholas, is doing both. Per Scholas trains testers to be contextual thinkers from the beginning and to bring the “manufacturing no more” values to their future work places. In the process, they are seeding a new trend that has the potential to shift old biases and pull the entire industry out of a pretty deep rut.

Let us hear your thoughts, biases, experience, frustration, etc. about your own work as a tester or test manager in the comment section. And stay tuned for the full video interview with Keith Klain, as well as an exciting new video series SmartBear is producing with him (think leaders in the software testing industry… debating).

See also:


By submitting this form, you agree to our
Terms of Use and Privacy Policy

Thanks for Subscribing

Keep an eye on your inbox for more great content.

Continue Reading

Add a little SmartBear to your life

Stay on top of your Software game with the latest developer tips, best practices and news, delivered straight to your inbox