Exploratory vs. Scripted Testing: One or the Other, or Both?
What is Exploratory Testing?
In the increasingly technical world that we all live in, the need for precise measures and a "five nines" mentality toward everything is growing at a rapid pace. The old adage “If it can’t be measured, it can’t be managed” takes on new meaning when analytics and dashboards rule the day.
Sounds a little uptight, doesn’t it? Where’s the ability to loosen one’s collar or let one’s hair down? In the world of software testing, exploratory testing offers a place where structure and process give way to intuition and gut feel, at least to some degree. Needless to say, it makes some people downright uncomfortable.
Cem Kaner coined the term exploratory testing. He has defined the practice as "a style of software testing that emphasizes the personal freedom and responsibility of the individual tester to continually optimize the quality of his/her work by treating test-related learning, test design, test execution, and test result interpretation as mutually supportive activities that run in parallel throughout the project." In layman’s terms, it could be said that exploratory testing allows for testers to learn on the fly, adjust their tests accordingly and, in that process, uncover software issues that are often unanticipated. The net result is an evolving picture of the software that better mirrors a user’s experience versus that of those who created it.
Feel free to stack Kaner’s definition up against the myriad of variations of that theme, which consist of subtle tweaks or complete rewrites of the original one. When you are done, you have the world of exploratory testing. This world has a bit of a Wild West feel to it, much like the practice itself.
This begs the question: if it’s so controversial (or at the very least inflammatory), then why is it being viewed as being so important as compared to other forms of testing? This merits a quick look at exploratory testing against its more ‘traditional’ counterpart, scripted testing.
How is Exploratory Testing Different from Scripted Testing?
What makes exploratory testing so troubling to some? The main reason is that it’s not running from a script. It’s more free form, although not to the point of being ad hoc.
Scripted testing follows a path that is written by the tester themselves or someone else. The script includes test cases and test steps that are documented. There can be no deviation from the path laid out in the script. The tester’s job in a scripted environment is to color within the lines only, follow each instruction to the nth degree and report on the findings.
Exploratory tests, on the other hand, rely on the tester to develop the testing path ‘on the go’. It’s not exactly a testing ‘free-for-all’ but it allows the tester to go where an end user might go rather than where a script tells them to go.
Both methodologies have their shortcomings. Scripted testing can simply lead a tester to a desired result. I liken it to a classroom teacher who teaches to the test. They are not so much concerned about what is learned by the student as much as they are concerned that the student produces a desired result. This type of approach can severely limit learning in a testing environment. Many potential problems that real-world users of a program might encounter may never fall into the scope of what the tester is told to test. The results of this testing may be a success in the areas that the tester was directed to explore but could leave many stones unturned. The result can be a buggy release that makes the end user scratch their head in disappointment.
Exploratory testing, on the other hand, gives software designers and developers the chance to see how their product may be used ‘in the wild’. While not without its restraints, exploratory testing can touch upon areas never considered by those who birthed the program. Like anything else in life, if you are too close to something you often miss even the most obvious issues and concerns. Being too close or familiar with a project lends to missing the details. Exploratory testing can help project owners see their work through a fresh set of lenses and give them insights that their own scripted tests would have missed completely.
Does that mean that exploratory testing is perfect and the answer to every software manufacturers’ dreams? Not at all. In fact, while exploratory testing may uncover the completely unexpected shortcoming of a program it could also miss the simplest problem because the tester simply never got there. Documentation in an exploratory environment can be found lacking in some instances as well.
So what’s the solution? In a perfect world we would see a mixture of both scripted and exploratory testing applied to a piece of software. This would give the best of both worlds and allow the design team to see as many angles as possible, thus making the release as bulletproof as possible.
Well, not sure about you but we found long ago that perfect doesn’t exist. Most companies are not afforded the luxury of doing a balanced blend of both. Often the decision to go one way or another must be made due to time and capital constraints.
With that said, let’s take a look at a few ways to help give exploratory testing a level of structure while allowing it to maintain the freedom that makes it an effective testing methodology.
How can you create plans that focus your exploration while remaining flexible?
Striking a balance between the scripted world of testing and its exploratory counterpart is not easy. In order for either side to keep its integrity, compromise is necessary. As with most arguments where both sides feel they are the right choice, this can be easier said than done. Let's look at a quick overview of three different options that exploratory testers offer to help keep testers ‘on task’ while still allowing the chance to run after areas that look like they need attention.
1. Test Charters
Charters are an important element that give shape and form to an exploratory testing model. Charters simply lay out the goals of a test session. They are fluid in that they can change, be removed, or added to as the testing progresses.
In a talk by Adam Yuret, he describes the structure of a charter by putting your charter in the following format:
As a tester, I want to (blank) in order to (blank).
Some exploratory testers may start to feel some restrictions when any format is applied to their tests. Unless there are some parameters, testers are in danger of becoming an ad hoc tester, which has its place for some, but can be so open ended that the tester can even have a hard time getting started.
In essence, test charters give testers direction without restricting where they can go. Charters define expectations without forcing the test to a specific result. They provide structure where none is wanted. In other words, they are a bit paradoxical but a necessary step to help testers and their clients ensure that there was a thorough test done that helped uncover bugs and issues that move the project forward.
2. Session-based Testing
In 2000, Jonathan and James Bach developed the practice of session based testing. It can be defined as “a software test method that aims to combine accountability and exploratory testing to provide rapid defect discovery, creative on-the-fly test design, management control and metrics reporting. The method can also be used in conjunction with scenario testing.”
For those who are used to a scripted test which allows for nothing more than the tester following the steps of the script, session based testing allows for a degree of structure that essentially gives assurance of what kind of work will be done and how it will be reported. Bach uses the term sessions to mimic that of someone recording in a recording studio in sessions. There is a time definition for each session--some say it could last a mere 20 minutes while the other extreme can provide for sessions lasting hours.
Many believe that the longer sessions go, the law of diminishing returns can take hold. The longer the tester is working in a session the more likely that tester will get too far away from what was defined as the desired area to be covered.
The desired result of session based testing is to deliver on the parameters of the test charters but allow for those charters to evolve as the tester moves through their process. In the end, session based testing allows for a structured delivery of results through session reporting.
The session is then debriefed using Bach’s PROOF method, which the testing manager goes through a process to ensure the findings are reported completely. The steps are:
Past. What happened during the session?
Results. What was achieved during the session?
Obstacles. What got in the way of good testing?
Outlook. What still needs to be done?
Feelings. How does the tester feel about all this?
Overloaded yet? Let’s look at one more way that an exploratory test can achieve the illusion of being limited in scope while keeping the freedom that makes exploratory testing so effective.
3. Mind Maps
I happened upon a definition of mind mapping in exploratory testing that is worth ‘copying’.
In his post “Demystifying Exploratory Testing” on stickyminds.com, Jonathan Kohl encourages the use of mind maps in exploratory testing. Mind mapping seems to coexist very well with exploratory testing since it's a rather free thinking exercise in and of itself. The point is to gain some structure so that when discoveries are made during exploratory tests they can be categorized and organized in a way that can assist those who are tasked with making the changes that result from the tests.
So to wrap up, what might be the most frustrating aspect of this subject is the sheer magnitude of it. The reality is that scripted testing and exploratory testing in the software world are philosophical issues at the core, and rather broad ones at that. Honestly there is a time and place for both techniques. There is no pat formula for when to use one or the other. Maybe the most important thing to remember is that, like with anything else in life, if taken to one extreme or another the results will be less than optimal. True discovery and progress occurs somewhere in the middle. It’s up to you to get there.
What are your thoughts on exploratory testing and scripted testing? Do you prefer one to the other? Let us know in a comment.