Thank you so much for your participation in the webinar Continuous Load Testing in Agile Environments by Michael Punsky. The webinar was well attended and we also had an excellent discussion with many insightful questions. The On-Demand recording of the presentation and the Q&A session can be found at the link above.
Before and during the event we received over 150 different questions. We tried to address as many of them as possible during the event, but some questions remained unanswered. After the event I discussed the remaining topics with the presenter Michael Punsky. Below you can see the notes from our discussion.
In general the bulk of the questions can be loosely grouped into three categories:
How to do load testing
How to integrate load testing into existing processes, including Agile; and
What tools to use.
How to do load testing and/or performance and stress testing?
Mike summarized the key activities representing basic guidelines for success in the following slide:
Here are some of the questions that we received during the event:
Question: How to automate load testing for 1000+ users?
Mike: The setup to run a test of 1000+ Virtual users is one controller, and a number of Remote Access Service (RAS) instances representing virtual users. The RAS instances can be on local machines, or they can be run from the cloud. For cloud testing, you must have an Amazon EC2 account.
Here is a video that describes how to use LoadComplete for Generating Load from the Cloud.
Question: How to perform load test from different geographical locations?
Mike: You can run from different geographical locations in the cloud by selecting cloud RAS instances that are from those geographies. North America East, North America West, South America, Asia..etc.
Question: How to check which webpage is not performing optimally?
Mike: When you run a test with a load testing tool like SmartBear LoadComplete, you are presented with a list of the 10 worst performing pages. That is the best place to start.
Question: How should we go about server side monitoring?
Mike: You can start by monitoring resource utilization in real-time as the load test is running. For automated testing as described in the webinar, you will need a resource utilization monitor that has the capability of time-stamping the events so that you can correlate resource utilization with the load generated during the test. LoadComplete provides such a capability.
Question: When measuring application response times during performance testing what kind of statistics do you use in results analysis?
Mike: Statistics I always check first are: Time to first byte, Time to Last byte, and page load time.
Process related load testing questions, including Agile
This part of the discussions seems to be dominated by two sub-topics. One sub-topic is more closely related to the Agile development process and how to integrate load testing in daily sprints and the other is more related to doing load testing in an agile fashion, but not necessarily within a specific process like Scrum.
Question: What is a better approach in Agile: to run service level testcases or for each API? In another way, Is it a good idea to "instrument" API function test suite with load test framework?
Goran: It is hard to provide a universal answer to this question because it really depends on the complexity of the system, importance of performance and risk involved. How much performance testing do you do today? If the answer is none until the deployment in the staging environment, or production, then there is an opportunity to do more. As Mike mentioned in the presentation - start small.
Question: How often to load test? During each sprint or only hardening sprints? As part of continuous build and test?
Mike: The best way to approach load testing is to test continuously, in regular intervals. Performance engineers should be plugged in at the very beginning of each sprint and review any performance deviations that get detected during continuous testing.
Question: How do you create the conditions in a test environment, very similar to the production environment?
Mike: General guidance is to try to reproduce the hardware configuration as closely as allowed by the budget. If your budget does not allow for an exact match, then try to have a scaled version of production in staging.
Question: How to convince the business that your test results are valid?
Mike: The best way is by taking action based on the test results, and then retesting to show the improvement. Continuous testing will help business see the value of testing and their investments for improvement of quality.
Question: How about SOA environments with frequent changes at the service level, or when attempting to test in environments that are also being utilized by other manual/automation teams to complete their testing?
Goran: In many ways challenges relative to continuous performance testing are similar to the challenges of continuous functional testing. If you are able to maintain a working version of your application and have an automated test environment performance testing - be it on the web services level, or at the application level, it would be similar.
Question: What measures do we need to take to make sure stress testing is matching with production load?
Mike: The best way is to have good analytics and to refer to analytics when designing your tests.
Finally an area that we intentionally did not cover in the talk because we wanted to focus on the challenges and guidance with solutions and more efficient use of load testing to ensure higher levels of quality:
Recommended Load Testing Tools
Even though the main topic of the webinar was focused on testing methods, strategies and best practices, roughly 20% of the questions were about tools. Here are some examples:
Question: What free load testing tool would you recommend?
Goran: This is a very common question, especially when deciding how to fit a new activity in the existing workflow. I would actually advise away from free tools because very quickly after learning the tool you may find yourself in need for additional support and training. For me personally, even more importantly, if you are successful with a pilot project you may want to introduce the solution to other people within your organization who may not have the same skill level, or who are not willing to put up with little quirks that you are ready to accept on a day-to-day basis. Instead of going with a free tool I would recommend adding a couple of commercial solutions to your list and evaluate them. You can request an extended evaluation license if you need more time. That way you can give yourself enough time to evaluate the tool against your needs and be in a better position for future activities.
Question: What are some of the alternatives to LoadRunner?
Goran: There are a number of good tools out there. As a vendor of performance testing tools I'd like to highlight LoadComplete which allows you to test web applications through front-end load generation either from available resources, or from the Amazon Cloud (currently in Beta). For testing of back-end elements of web applications you can take a look at loadUI Pro which is a commercial version of the open source tool loadUI. loadUI Pro will allow you to reuse soapUI tests for automation and target APIs and SOA components.
This webinar was an excellent starting point into discussion about load testing in agile environments. We want to hear more of your thoughts on this topic. Go ahead and share your experiences in the comments below.