The Risks of Load Testing with Third-Party Monitors
One thing you should be striving for when you're load testing is a clear cause-and-effect, or causality. If you are unfamiliar with this term, it simply means that you want to make sure that the results you are seeing are due to whatever it is you are consciously throwing at your service or website.
If your load testing results suddenly show you a spike in performance, you want to be able to backtrack easily to find what caused it. This is one of the major reasons why having monitors set up at your different performance points is important. If your load testing setup only lets you see the performance from a user perspective, in terms of load time and response rate, it becomes a guessing game to try and figure out what’s actually causing certain behaviors. You might end up investing hard-earned dollars into more CPU, RAM and server hardware, when the actual problem might have been hiding in poor database-configuration all along.
Being able to dig down and see exactly how your tests are affecting your Web server, database, OS and application server allows you to be more confident regarding the causality. This could, in the end, save you time and unnecessary investments.
The Risks of Third-Party Monitors
Just because you have monitors set up doesn't mean you can kick back and relax just yet. If you’re using one tool to monitor your load tests and another to generate the load for them, you’re back in that causality problem.
We noticed that some of the users of our API load testing tool, LoadUI, were using external, or third-party, monitors to track the effects on their server. While using third-party monitors is better than using no monitors at all, they do come with a whole new suite of risks.
First of all, you (or your team) need to learn an additional tool. This takes time and causes issues with knowledge-sharing across the team or organization. A better solution is to find one tool that allows you to both create the load and monitor the results. This has the added benefit of only spending the time to master a single tool, and using that tool to its fullest potential. In the case of the previously-mentioned LoadUI, you lose a large part of the exploratory load testing workflow when you need to switch back and forth between tools.
Secondly, and perhaps more importantly, you might be tricking yourself into seeing causalities when, in fact, there are external factors playing a part. Third-party monitors usually force you to install software on the server. This, in itself, will affect the performance of your server. That monitor will need to share things like CPU and RAM with the applications and services you want to load test, which can skew the results.
Instead, if you delimit the amount of processes running on your server by not installing additional third-party monitors you will not only decrease the amount of work your server will need to pull through, you'll also have clearer causalities between the server’s performance and your load tests. If there’s nothing else that can play a part in affecting the performance, you can be confident that it’s your load tests that are causing the results.
Also, because they’re two different tools in two different environments, the timing between the load generation might not totally sync up. By using one load testing tool to do both parts, you can be confident that what you’re seeing in the server performance was caused by load generated at the same time. Having the ability to overlay the load generation on the same graph as the server performance metrics makes finding correlations and bottlenecks much easier.
With LoadUI Pro you get a fully features load testing tool for your APIs. It includes built-in server monitoring, and doesn’t require you to install software on your server. It’s also fully integrated with SoapUI. Try out a 14-day free trial today.