Performance Testing: The “External Content” Challenge
About a dozen years ago, I had an argument with a client. Actually, that is an understatement. It was an all-out fight that nearly came to blows. For those of you who know me, this may not surprise you overmuch. What is somewhat surprising (or at least disappointing) is what caused the fight. You see, this client wanted me to include “clicking all of the links and loading all the pages” as part of the performance test – and while this might not sound terribly unreasonable on the surface, many of the links he wanted me to click, and pages he wanted me to time the loading of, were outbound links.
In other words, he wanted me to load test the home pages of sites that his company had no control over or relationship to (can you say “Denial of Service Attack?”)
And why did he want me to do this? Because he was absolutely convinced that when a user clicks one of those links, it sends some instruction to his webserver for it to retrieve the page and serve it to the user (while this scenario is possible, trust me, this was not the case). Therefore, he was convinced that users clicking on those external links—regular old <a href> outbound links, not buttons, not scripts, not even a “user x clicked link y” tracking message being triggered and returned—were going to generate so much overhead on his hardware as to take the site down.
Three weeks, no fewer than six demonstrations, and four “expert consults” from both his developers and the lead architect from the company I was working for later, he finally relented. I don’t think he either understood or believed us, but he relented.
Luckily, I’ve not had this particular problem since, but I’ve had what is essentially the inverse argument regarding third-party ads, tracking scripts, and other inbound content. Of course, these items don’t cause additional load on your servers, but they do have a noticeable impact on the end-user’s real and perceived response time.
Worse, is that this content is notoriously difficult to script and maintain OR remove from the load test. It is even more difficult to set up a test so that only a small percentage of your simulated users load this content, which is what you need to do if you want to time the delta between the responsiveness of what is under your control and what is not under your control.
Of course, the most useful thing to do has historically been the most challenging, which is to execute some tests with the third-party content and some tests without so you can assess their impact. In fact, what that often leads to is the desire to run more tests with specific subsets of the third-party content enabled or disabled. Unfortunately, with most load generators, this process is somewhere between cumbersome and impossible given time and resource constraints.
This is why SmartBear has released what we call the “Host Configurator” with LoadUI Web Pro 2.9. With the Host Configurator, this process amounts to nothing more than identifying the base Host Name of the content you wish to include or exclude for a particular test and check or uncheck the corresponding box for the Host or Hosts you wish to include or exclude from your test. This amounts to enabling you to run a far greater variety of tests, collect and analyze a greater variety of data, and make far more informed decisions about which or how much third-party content to include in your website or Web app without blowing the schedule.
If you have felt the pain of trying to manage which third-party content to include in your load test like I have, I suggest you give LoadUI Web 2.9 a try.