Workload Modeling and Profiles for Load Testing
Any load testing project should start with the development of a model for user workload that an application receives. This should take into consideration various performance aspects of the application and the infrastructure that a given workload will impact. A workload profile is a key component of such a model. Depending on the type and goals of a load test, one or more profiles may be appropriate. Choosing the workload profiles representative of anticipated real load over time (whether it is an everyday usage scenario or a high peak) results in more accurate answers to the “main questions of load testing” such as, “Will my site support N users performing a search at the same time?” and “What is the highest number of users that my site will support – while remaining within specified quality and performance guidelines?”
Workload modeling identifies one or more workload profiles to be simulated against the tested application. The workload model then attempts to approximate real life usage scenario and includes different user types and characteristics.
To determine the workload for your application, consider the following questions:
What is the set of possible actions that a user can perform?
This depends on the specifics of your web-application. For a typical e-commerce application, consider the following actions:
- Connect to the home page.
- Log on to the application.
- Browse the product catalog.
- Search for specific products.
- Add products to the shopping cart.
- Validate and place an order.
- Log out from the application.
What are user profiles for the application? Will a single user profile be enough?
A user profile is a part of a load test model that is built around a typical use case, or a scenario, of a real user interacting with the tested application. For example, for an e-commerce site, there are groups of users who simply connect and browse the product catalog, and there are other users who log on, search for specific products, get price information, maybe add some selections to the wish list, and then log out, etc. Based on this approach, you can determine what the most relevant user profiles are for your load test, for example:
- Browse profile
- Search profile
- Place order profile
Note that if a load test employs multiple user profiles, it is best to debug each of the profiles separately. Once you’ve confirmed that each profile performs adequately on its own, you can begin combining multiple profiles into a single test.
What are the actions performed by a user representative of each profile?
For most cases, the user actions should be relatively obvious based on the use cases that were used to design the application. There may be some steps that are performed more than once, and such statistics could be approximated based on user modeling, or server logs for existing applications.
For example, let’s say that, on average, each user who buys products on your site purchases three products in a single session. The breakdown of the various actions in the user profile may include:
| Operation | |
|---|---|
| Connect to the home page | 1 |
| Log on to the application | 1 |
| Browse the product catalogue | 4 |
| Search for specific products | 2 |
| 3 | |
| Validate and place order | 1 |
| Log off from the application | 1 |
You should perform this analysis against all user profiles.
What is the average user “think time” between actions?
The time spent by the user between two consecutive actions on a page is called “think time” in load testing parlance. During “think time,” the user may be viewing the information displayed on a page or may be entering details such as addresses, credit card numbers, search parameters, etc.
Think time can vary depending on the complexity of the information on a page. For example, the think time for a logon page is less than the think time for an order placement page where a user must enter credit card details.
Most load testing tools utilizing browser-based recording will capture the actual delays between user actions. These values can later be changed, averaged between all requests, or substituted by automatically generated values with a given set of statistical parameters.
What is the expected ratio of user profile scenarios?
Combining different profiles allows you to create a more realistic simulation of the web traffic for scalability tests. The ratio of business actions performed by various user profiles can be estimated or measured. For instance, a typical usage pattern of an e-commerce application could be 77% of users browsing the site; 20% searching for particular products, and 3% placing an order. This is based on a conversion rate of 3 percent (your conversion rate may differ). The ratio depends on your site, your business and your users. It is important to have the ratio reflect the actual transaction on your site.
How does the number of users logged on to your site vary with time?
In an actual deployment, the number of users will vary by time of day, week of year, what is happening in the news, etc. Understanding and accounting for these variations will enable you to develop a better workload model.
A flower-delivery site may have a relatively modest steady load during the day with a ramp up towards close of business when people remember that they need to send flowers as they are leaving work. And then the site hits a super-high steady peak load during the week leading up to the Valentine’s Day. An online banking application may have a different usage pattern altogether.
For some load tests, getting the workload profile to match that variation on some scale is important, and for some it is not as much. Changes in load during the day should be factored into a long running performance test with the objective to “mock up” the actual load profile that the server will experience and analyze the metrics from the infrastructure. On the other hand, earlier in the development cycle, it may be easier to run steady/constant or step-wise increasing load as these profiles facilitate easier interpretation of results.
What is the maximum expected number of users logged in to your application?
This number represents the maximum operating capacity for your application. Simultaneous users are users who have active connections to the same web site at the same time. If you are developing a new application, the source of these numbers may be the application’s requirements document (which is based on the predicted usage of your application according to the best guess of your marketing department). For existing applications, you can identify the number of simultaneous users by analyzing your server logs and/or the information provided by your analytics service.
What is the duration for which the test needs to be executed?
The test duration depends on the specifics of your application and may range from 20 minutes to as long as a week. Consider the following scenarios:
- You need to test your login mechanism to ensure that a required number of users can log in to the site within a certain period of time. This scenario can be testing in a short period of time (15-20 minutes) and the results can be extrapolated for longer periods. If 100 users can log in within 15 minutes, it can be extrapolated that 400 users can log in within an hour.
- You may run tests continuously for an extended period of time (days, and possibly weeks) with a constant load to observe how your application performs over the long haul. This identifies slow memory leaks that only become apparent over a long period of time.
Hopefully these recommendations help you in constructing a workload model that reflects the usage pattern of your web application, thus making your load tests much closer to real-world scenarios. I would like to acknowledge contributions to this blog post by multiple SmartBears, including Michael Punsky and our tireless technical writers group.