Pitfalls of Script-Based API Load Testing

  March 29, 2017

Think Thrice Before Committing to a Script-Based Solution for API Load Testing

If functional testing is the first step towards API quality, then load testing is the second. Yet, even though it’s the next logical step, teams testing APIs often struggle with the transition. This is often because teams end up adopting script-based API load testing solutions – but it doesn’t have to be that difficult.

Scripts add process overhead to API testing and prevent teams from easily reusing their functional tests as load tests. To build robust load tests, teams must learn concepts like load test patterns, virtual users, distributed testing and server monitoring. This by itself is a lot of work. Introducing scripts on top of this adds a need for specialized training, hence requiring a significant resource commitment from the team.

Solutions that fall in the category of script-based load testing include LoadRunner, by HP. LoadRunner has been extended from frontend load testing to perform API load testing as well, but using such a tool creates issues for teams in the following three areas.

Load Test Creation

As per the recent developer survey conducted by StackOverflow, only 3.5% of developers are working as QA Engineers. This might be indicative of the shift left trend, but more so it demonstrates the squeeze on QA focused coding skillsets and highlights the fact that scripting and coding for QA is done by a small number of software professionals, which makes it harder and expensive to hire them.

If you are thinking of training a tester/developer in house, then be prepared for two major expenses: first is the initial training costs; and second is the initial ramp-up time that is lost when the users are still learning to tests with scripts. This is confirmed through a simple analysis of long tailed keywords around "LoadRunner api" on Google.

The results show that there is a high volume of searches for script functions like "web_custom_request." This indicates that testers using LoadRunner are struggling with cryptic function names and spending a lot of time figuring out solutions to their scripting woes online rather than spending time creating and executing load tests.

[caption id="attachment_23266" align="alignnone" width="556"]Screen Shot 2017-03-29 at 9.29.18 AM A sample of scripting method searches done by users of LoadRunner on Google[/caption]


Load Tests Maintenance

Maintaining scripts is like maintaining code, except that often there is no hierarchy that comes with programming languages like Java, .Net and C++. These languages provide object-oriented structure and naming conventions. This implies teams need to build their own custom structure and framework, in order to work with a multitude of scripts and manage them as their numbers balloon.

This structure also brings code maintainability issues to the forefront, i.e. making it easy to comprehend what one tester wrote in a script to another tester. This requires testing leads to enforce guidelines around writing comments in the code. If naming conventions for scripting are not followed, the whole scripting setup becomes incomprehensible to a new tester joining the team.

Teams working on scripts often spend a lot of time going through code written by other testers and understanding the usage of variables and logic. Bottom-line: It’s very easy for a tester to write scripts in their personal style, which might not be easy for the rest of the team to comprehend. This adds additional overhead and often reduces the team’s productivity.

Tool Lock-in

Scripting tools force teams to build practices around code control, while also managing the complexity that comes packaged with scripting. Over the course of time, as load tests increase in number and teams build processes for version management of scripts, some may even require scripts to be reviewed before being used for correctness and to ensure naming conventions are being followed. These activities result in increased resource allotment to overhead projects like reviewing scripts. This has the undesirable side effect of creating a lock-in of practices and processes with the tool.

Another issue that surfaces is the need to have all of your tests in the form of scripts, which are incompatible with other platforms. In addition, your team might not be willing to adopt a new tool, as scripting is their key skill and they believe the adoption of other tools may make their skills redundant. This unwillingness by team members to get behind the management’s decision to adopt and change tools is often a major problem for teams that use script-based tools like LoadRunner.

For some testing use cases, there is often no other alternative to a script-based solution – but for API load testing, why would you use a script-based solution when an easy-to-use, scriptless load testing tool already exists?

User Friendly Solutions Are The Way To Go

Point and click solutions like LoadUI NG Pro provide an easy way for testers to transfer their existing skills to load testing. Load UI Pro has a very shallow learning curve that allows tester to easily create load tests with intuitive wizards at the click of a mouse. They can also reuse their functional tests as load tests. No specialized skill is required to work with LoadUI NG Pro – anyone on the team can start working with the tool in minutes.

This tool has been built from the ground-up for API load testing. Thus, even the most complicated testing scenarios can be created on the user interface, without any need for scripts. The test cases are visible in a pre-configured hierarchy and the composite project feature provides a way for teams to share work on a single project.

LoadUI NG Pro also comes with some advanced API load testing features, like distributed load testing and server monitoring, to provide you with a holistic view of the performance of your application. For testers who want very granular control over their tests, LoadUI NG Pro provides groovy scripting hooks too. Compared to LoadRunner, LoadUI NG Pro is also cheaper and has lower total cost of ownership as teams work faster and there are no resource commitments required for overheads like script maintenance.