In the world of performance testing, it's important to understand the various types of testing, what they consist of and how they can benefit your applications. Just as carpenters need the right tools to build a house, performance testers need the right test to accurately analyze web performance.
This blog post is designed to help you understand exactly which tools you'll need for the job. I will list a few types of testing under the performance testing umbrella, give a brief definition of each type, and describe the problem that it can solve or the benefit that it can provide.
7 Types of Web Performance Tests
1. Performance Test: A performance test is any test that measures stability, performance, scalability and/or throughput of your web application(s).
2. Capacity Test: A capacity test is a test to determine how many users your application can handle before either performance or stability becomes unacceptable. By knowing the number of users your application can handle “successfully”, you will have better visibility into events that might push your site beyond its limitations. This is a way to avoid potential problems in the future.
3. Load Test: A load test consists of applying load to an application and measuring the results. The load may or may not be at the high end of application capacity. These tests can help determine normal performance metrics. By using iterative testing, you can determine whether new code has helped or hurt performance.
4. Stress Test: A stress test is a test that pushes an application beyond normal load conditions. When you push your application to the extreme, you will see which components fail first. Making these components more robust, or efficient, will help determine new thresholds.
5. Soak Test: A soak test is a long-running test that is used to determine application performance and/or stability over time. An application may work well for an hour or two, and then start to experience issues. These tests are especially useful when trying to track down memory leaks or corruption.
6. Component Test: Testing a discrete component of your application requires a component test. Examples might include a search function, a file upload, a chat feature, an email function, or a 3rd-party component like a shopping cart.
7. Smoke Test: A smoke test is a test run under very low load that merely shows that the application works as expected. The term originated in the electronics industry and refers to the application of power to an electronic component. If smoke is generated, the test fails and no further testing is necessary until the simplest test passes successfully. For example, there may be correlation issues with your scenario or script – if you can run a single user test successfully, the scenario is sound. It is a best practice to initate one of these “verification” runs before running larger tests to ensure that the test is valid.
How These Testing Types Fit Into Your Overall Test Cycle
It's the Journey, Not the Destination: The first concept that a performance tester should embrace is that testing is not an event, but rather a process. You should test continuously, and build testing into every aspect of your development cycle. It is an iterative process, meaning that tests will be repeated numerous times and actions will be taken based on the results of your testing. Then, the test is repeated, generating new, actionable, results. Different types of testing will plug into the development process at different points. An application’s overall success will be dependent on the information gleaned from your testing.
During early stages of development: Smoke testing will be used to validate that the individual components work as expected and that they integrate into the application in a functionally correct way. These tests involve the use of single or very few virtual users and will merely be used to determine whether an application does or does not work.
Component Testing will be used to validate and measure the results of the individual components of your application. These components include searches, shopping carts, database inserts, logins, file uploads, downloads and connections to third party components, among others. Components will be tested using few virtual users.
During development: Performance testing will be used to determine the performance characteristics of your application. This may involve heavier use of virtual users, but should not be taken to an extreme. The point of the performance testing process is to determine how the application performs under normal or low load conditions. Are there places where performance should improve? What are the bottlenecks in your application and how can you tune those aspects to perform better?
In later stages of development: After you have tuned your individual components and tested your application for performance, it is time to test under normal expected load. You will run tests using the expected number of users, and see how the application performs. Additionally, you will see if errors are generated. How stable is your application? If your application cannot handle expected load, you will need to make changes (refactor code, tune or add additional hardware, tune your queries and indexes, etc.). Then, you will need to retest the application to check for improvement.
Pre-deployment: Prior to launching your application, you will need to run successful load tests at expected traffic levels. Additionally, you will need to run soak testing to determine how your application runs under load for an extended period of time. You will look for things like memory leaks and memory corruption. If these things are found, they should be remediated.
Finally, you will run a stress test and determine:
- How much load your application can handle before it fails
- Which parts of your application fail first
- How does your application recover after failure? Once the load drops, does the app recover gracefully, or do you need to restart the application server? The database? The web servers?
Post deployment: You will need to run tests periodically to determine if performance has degraded. Even if your code has not changed, the amount of data used by the application might, and performance could suffer as a result.
If an upcoming event could generate increased traffic to your site, you will need to ensure that you can handle it successfully. Items like press releases, speeches, product announcements, and advertising campaigns are wildcards that can increase your traffic radically in a short period of time. Being able to support a level of traffic higher than your normal heavy traffic days is imperative.
Lastly, any new revisions of your application should go through the same process as outlined above. The more you know about the way your application behaves, the better prepared you will be for potential bumps in the road. It will make your applications more successful and your career as a performance tester a little bit easier!
How have you been using web performance testing in your testing cycle? Please share your ideas, experiences and comments below.