Should Exploratory Load Testing Be Part of Your Process?
If you do a Google search of the concept of exploratory load testing for software, you quickly realize that there isn’t as much material about it as you might think. Why? Well, it’s not an easy concept to grasp, quite honestly. Sure, some may say “I understand it!” but when you delve into their "explanation," they are actually talking about exploratory software testing. We have written plenty about exploratory software testing here at SmartBear, but when you add one simple word -- load -- well, things change a bit.
Why is that? It could be that it is simply difficult to define cleanly. Not to brag, but one of the better looks at exploratory load testing is right here on the SmartBear blog. Written by Ole Lensmar, he looks at the idea of exploratory testing which is intended to help uncover issues in a software program based on the mimicking of what an actual user might do. In other words, you put the software through the ringer from a customer's point of view rather than something that has been scripted, or designed, to examine a particular area. In the process there are oftentimes scenarios or bugs that are revealed that may have been missed in a scripted environment. There is real value there.
Now take that mindset and channel it towards load testing your software. Exploratory software testing relates to the users experience as an individual but is not designed to see if the software can ‘take a punch’. In other words, will your software shut down if too many people want to do something in the program at once (think SaaS delivery)? If too many people hit your shopping cart as they react to a special offer will your program collapse under the load? If you put in too much data for a particular function will you bring the program to its knees? Where might there be bottlenecks that could cause the program to underperform?
Scripted load tests can be designed to look for specific areas of trouble, but as with scripted testing of the program in general, it can miss some of the nuanced situations that might arise outside of a defined scope for testing. An exploratory load test can build upon itself by identifying other areas to test as observations and results occur during the course of the test.
That’s all well and good but how is this accomplished? What should a tester be thinking when doing an exploratory load test? Good question since the answer fully depends on the type of software or web application that is being tested. What may be critical is the ability of the tester to think "on their toes" and respond to results in a way that works toward uncovering other issues or concerns.
Strong exploratory load testing will be the result of the tester working a scenario, while using load testing tools that will see how stretching the capabilities of the software or web application. Some areas to consider include adding and removing transactions and virtual users, increasing or decreasing load, changing load characteristics, and monitoring server performance and client response times.
Does this mean that you ignore traditional load testing in favor of exploratory tests? Not at all. We have discussed the idea of being balanced in testing approaches in the past. Combining exploratory load testing techniques and tactics in conjunction with scripted load testing is highly recommended. No one way covers all of the bases. It’s just that exploratory testing uncovers those bases that, until they are discovered, may not have been known until your end users were using the program or application as they do -- which is to say, in ways developers often never imagine. As Lensmar puts it:
“Of course LoadUI allows you to still run the traditional “generate X transactions per Y from these Z agents for U minutes” test (and you should!) – But I firmly believe that both approaches are necessary to get a realistic “performance profile” of your application – be it Web or API.”
Not to oversimplify this matter but what is truly required by load testers in an exploratory environment is to keep a very open mind and imagine where extreme loads (number of users, CRUD transactions, network inefficiencies) might hobble a program or application and bring it screeching to a halt. One way to do this can be Gorilla testing.
In functional testing this approach heavily goes after one area of software functionality (often by people with little or no experience in the software or even testing itself) to just ‘hit the wrong keys’ again and again. An example might be discovering what happens when a form is entered several times by repeatedly hitting the call to action button. Does the program throw back an error message to the user? Does it submit the form multiple times? You get the idea.
In the exploratory load testing sense, however, it takes a different form where the tester is asking questions along the lines of “What happens when I turn this knob, add this component or pull this lever?” In the end, it carries the same philosophical approach because exploratory testing, in general, is truly about going where no user has gone before. When there are no true rules, the learnings are often unanticipated yet quite valuable.
In all, exploratory load testing of software can seem a bit ambiguous. Heck, the idea of exploratory testing of software itself is the same way. It’s one of those things that you know makes sense but can be difficult to put into words. It’s this very nature that creates some confusion and more than a little trepidation by project managers when it comes to load testing a program.
Our recommendation? Relax a little and maybe spend some time outside the box. What can it hurt? Oh and by the way, we’re here to help you get over the exploratory load testing hurdles with our LoadUI Pro and LoadUI Web products. We’ll remove some of the fuzzy edges and let you get the results you need.