Performance Testing: The Environments Challenge
Test and Monitor | Posted November 05, 2013

Over the years, I’ve seen all kinds of challenges facing performance testers. Some due to technical constraints, some due to organizational constraints, and some due to situations so complex it was difficult to pinpoint the cause. But, of all the questions people have brought to me, two stand out as by far the most common:

  1. “What tool should I use for {situation X}?”
  2. “How do I handle performance test environment challenge {X}?

The tools question is far too context specific to address usefully in this format. Besides, I have nothing to say on the topic that I haven’t already said in publicly available blogs, articles, webinars and books. On the performance test environment challenge topic, however, I believe there are some new things to discuss. Why? Because virtualization, the cloud, and tool support have become commonplace enough to fundamentally change my core recommendations – which I hope you find valuable.

So what performance test environment challenge am I referring to?  I think the following 5 questions represent the challenge better than a narrative description:

“What do I do when…

  • … I have to script against one environment, but run my tests against another?”
  • … my performance test environment doesn’t match production?”
  • … the IP/URI of the performance test environment is constantly changing?”
  • … I’m asked to run a “quick test” against a different environment?”
  • … I need to simulate different users going to different URIs to simulate geographic (or other) diversity (e.g.,,”

Certainly, there are other performance test environment challenges (such as “How do I deal with having to share a database with the functional test team?”), but as with the “What tool…?” challenge, I don’t really have anything new or exciting to add to what I’ve already said/written on the topic.

If you’re not a performance tester, you’re probably wondering what the big deal is if it’s not about test data… And that’s a fair question.

Unlike manual testing, or even functional test automation—where you generally only need to change the entry IP, URI or URL in one place (frequently the browser address bar)—when you’re working with a load generator and you want to test against a different environment, it’s not quite that easy. Since load generators work almost exclusively on a protocol level, to point a test at a different environment means changing the IP, URI or URL, not in one place, not in one place per page, not even in one place for each object requested by a page, but between one and three (on average) places per object per page—and with typical webpages these days averaging well over 40 objects per page, simply “pointing to a new environment” can very quickly add up to over 100 changes per page for every page you’ve recorded and plan to use in the test.

Okay, that can quickly add up to a big number, but that’s what search & replace is for, right? Unfortunately, search & replace isn’t as much help as you might think.

Firstly, not all load generation tools even enable search & replace at the correct level of the script to be of much use (and I can’t think of any that have a “global” search & replace that works on all the scripts at once.)

Secondly, for those that do have search & replace, the IP, URI, or URL that you want to replace is not represented the same way everywhere. Sometimes it includes the HTTP://, sometimes it doesn’t. Sometimes it includes the trailing slash, sometimes it doesn’t.  Sometimes the slashes and dots are encoded, sometimes they aren’t.

You’ll often find what you want to change in the GET & POST blocks, but some of those point at a different host entirely (by design). Sometimes you’ll find what you need to change in REFERER: strings, sometimes you won’t, and sometimes it will be there but won’t matter. Sometimes you’ll find it in Cookies, SessionIDs, ViewStates, and/or any of a million other security or session tracking methods out there.

So, the obvious thing to do is use Search & Replace to replace all of those entries with a super-cool variable (or series of variables, so you don’t need completely different variables for the encoded, unencoded, and partial versions of the base IP, URL, or URI you’re interested in). Of course, now you have to deal with the scope issue, which comes into play because most load generators treat each script as a more or less separate program until it’s compiled into an executable at runtime.

Naturally, that means that you still need to change this super-cool variable once for each script, unless you’re working with a tool that allows you to set global variables. Or maybe you’ll decide to read the IP, URI, or URL in from a file like you do for usernames and passwords. Or maybe you’ll come up with beautiful hack to work around whatever complication your particular tool has thrown in your way, but no matter how you overcome this “way-more-complex-than-it-should-be” task, you’ve still put in way more time than it would have taken to run the necessary or requested test in the first place.

If it’s that hard, you ask, why not simply say “we only run load tests in the load test environment”? The truth is, that may individuals and organizations have done exactly that, but that causes problems of its own. Consider these. If you are only able/willing to generate load against a single environment per application, you’ve just eliminated your ability to:

  • Help developers optimize load-sensitive aspects of the system until that part of the code reaches the load test environment (which is typically one of, if not the, last environment the code goes to prior to production).
  • Help administrators optimize load-sensitive configuration settings prior to load testing.
  • Generate some load against the functional test environment so the functional testers can check for things like timeouts, data corruption, etc.
  • Run a quick battery of load tests against production while the system is down for the upgrade to see if the build got promoted correctly and if the numbers you got from your environment appear to have any correlation to what you get on production hardware.
  • To run low-load scenarios against a dev’s box while they are trying to optimize something particularly tricky so they can get it right (or at least close) before integrating that code into the build.

Clearly, I’ve spent a lot of time struggling with this challenge. And by a lot I mean probably 20% of the time I’ve spent in my career scripting & executing performance tests can be attributed to doing what is necessary to execute those scripts against differing environments—and probably another 25% of that time pleading with vendors to build an easy way to do this into their tool.

Well, at least one vendor has finally done it. If you’re a performance tester who knows the pain I’m talking about, check out LoadUIWeb 2.9’s “Host Configurator” feature. It allows you to swap environments (the first time) by doing little more than copy/pasting the new host URL into a text box.

After that, anytime you want to test against that host, you only need to check the box with the host you want to run your test against and you’re ready to go. Just like that. In less time than it took me to write this paragraph.

Pretty cool, huh?

If you like that, wait until my next post where I tell you what else you can do with the Host Configurator!

See Also:

[dfads params='groups=930&limit=1&orderby=random']

[dfads params='groups=937&limit=1&orderby=random']