Posted September 19, 2017
Cloud and Parallel Testing Trends in the SmartBear 2017 State of Testing Survey
Over 5,000 professionals in Software Development, QA, and Testing responded for the SmartBear 2017 State of Software Testing survey. One of the areas that received interesting feedback was the trend of parallel and cloud testing.
Parallel testing has emerged as a way for testers run the same scripts in multiple environments concurrently, allowing them to increase coverage while cutting testing time. Seventy-six (76) percent of respondents are parallel testing with the highest percentage running two to five UI tests in parallel at 35 percent. However, there’s still a number of organizations who don’t leverage parallel testing at all with 23 percent of respondents reporting that they do not run any UI tests in parallel.
Participants who do web testing (desktop or mobile) are more likely to conduct parallel testing at 77 percent than those who do not at 66, and participants responsible for end-to-end testing are 10 percent more likely to do parallel testing than those focused on just UI or API.
We also gathered information about the use of cloud testing. Cloud testing services provide a remote environment for teams to access a wide range of machines with varying browsers and operating systems that are regularly maintained and updated for testing. Cloud testing often goes hand-in-hand with parallel testing since it provides an ideal environment to run multiple tests in the most recent browsers, devices, and operating systems from one location without maintaining multiple VMs or a device lab.
Thirty (30) percent of respondents reported running tests in the cloud, while 34 percent reported not running any. Testers who conduct parallel testing are also more likely to run tests in the cloud with 56 percent of respondents who reported doing some parallel testing also running some tests in the cloud, compared to those that don’t do any parallel testing where 0 percent of respondents use a cloud service.
While there are many teams and organizations leveraging the power of parallel and cloud testing, there is still an opportunity to utilize these methods further. Although the majority of teams understand the value in complementing the two testing methods, it seems there is a lack of people harnessing parallel testing in the first place. This means that only about 43 percent of total respondents are utilizing both parallel testing and cloud testing.
As browser, device, and operating system versions are released more often, web testing on one or two configurations will no longer be a complete enough solution to account for fragmented users. Instead, testing on a selection of machines will help evaluate whether a web application is acceptable to the majority of users.
Through parallel testing, teams and individuals can run a test script across multiple browsers, devices, and operating systems at the same time to avoid having to sequentially run the same test again and again on different machines. Cloud testing takes this a step further by giving on-demand access to thousands of configurations for parallel testing.
Additionally, many respondents don’t utilize these methods separately even though there are benefits to that as well. For example, cloud services are a helpful resource for tasks such as manual testing and debugging, or taking full-page screenshots, but no respondents reported using a cloud for testing if they are not running any parallel tests.
As testing processes must get faster and more accurate to keep up with developing trends that we’ve witnessed such as Agile, Continuous Integrations, and DevOps, it’s critical that teams utilize the tools and methods that will help them effectively contribute to these initiatives.
Organizations that apply practices such as parallel testing and cloud testing, or better yet utilize both with services such as CrossBrowserTesting, will find they able to enhance their daily testing processes to effect faster feedback and more rapid development and delivery cycles.