The rapid adoption of agile along with continuous delivery necessitates a shift towards higher levels of test automation. And while application development and delivery (AD&D) teams move towards increased test automation, they primarily go about creating and maintaining test environments in two ways:
- Using an In-house Infrastructure:
In this case, your operations or test team takes on the responsibility of building an in-house infrastructure themselves, including the servers, mobile and desktop devices, and integrations to CI tools.
- Using Cloud Virtualization Services:
The other way you can go about setting up a test infrastructure is by spinning up virtual machines using services such as Amazon EC2 and Rackspace.
Regardless of whether you are setting up an on-premise infrastructure or using cloud virtualization services such as Amazon EC2 or Rackspace, you’ll still be required to put in some manual efforts to build, maintain, upgrade, or teardown the test infrastructure.
The efficiency you gained by automating testing of desktop, mobile, and web applications can thereby be easily eroded when test environment management needs manual efforts. While your automated tests are waiting to be run, teams spend hours performing repetitive and redundant tasks like downloading the right browser version, waiting for the test machine to be available, or ensuring they have access the right operating system.
The challenge to test automation just increases when the test environments have huge discrepancies from production environment. Your end-users are probably using a different resolution, browser version, or an operating system version than the one you ran your tests on.
The result: Your test environments become the weakest link your test automation cycle. Learn how to overcome this hurdle.
Test Environment Management - Think about Scalability, Cost, Maintenance, and Ancillary Testing Services
1. Labor Cost:
Regardless of whether you are using an in-house infrastructure or a cloud virtualization service, there are few key factors that will define labor costs. First and foremost it is the time and effort your operations, test, or DBA team spends on maintaining, upgrading, or even tearing down test environments, database servers, and labs, among others.
While cloud virtualization services can help you overcome hardware up front, there is still a manual component involved in ensuring the right configurations are available when tests are being kicked off. Provisioning clean test environments often means spinning up new VMs with new configurations which can be time consuming and adds to the labor cost.
Either way, your test engineers, DBAs, or operations team can focus on other important problems that result in higher return.
2. Device Cost:
This represents the cost for maintaining and upgrading different desktop and mobile devices. Keeping systems and devices up-to-date can become challenging as teams grow and new environments (web browsers, operating system version, and resolutions) are added to the mix to improve coverage.
As the team grows, ensuring particular combinations of operating systems, resolutions, and browser versions are available for testing means adding new devices or spinning up a new VM. This cost can grow quickly when different operating system and browser needs to be kept up-to-date based on the latest release.
And if the testing team is not co-located, there can be some degree of lag while accessing these devices, which in turn impacts the productivity of the team.
3. Software License Cost:
Other than taking the responsibility of maintaining the desktop hardware, virtual machines, and other real devices, the DBA along with operation also needs to have host of other software licenses when managing environments using VMs or an in-house infrastructure. These licenses just add to the overall maintenance cost.
4. Parallel Execution:
The time it takes for your testing suite to complete a run can be drastically reduced by running UI tests in parallel. Ensuring your in-house test infrastructure or cloud virtualization service can handle parallel execution often requires extra configuration or setup steps.
Running tests in parallel can get time-consuming, especially when you are not able to dynamically scale devices as per your needs.
5. Debugging & Reporting:
Features such as videos, logs, and screen shots can be helpful once tests have been run to ensure they are debugged in real-time and so that you can see what’s going on with a build.
However, building an in-depth debugging mechanism in-house can be challenging. The additional time and effort you would spend while building in-house, real-time reports and dashboards for common visibility across team members could easily be spent on higher value items with immediate benefits.
6. Advanced Testing Features:
Getting a deeper visibility into your testing efforts necessitates having additional advanced testing features, which are primarily lacking with an in-house infrastructure or even cloud virtualization services. Primarily because none of these are designed with QA teams as the target market segment. Take the following value add feature as an example:
With increased market fragmentation resulting from screen size, device type, and hardware specifications, testing can easily get messy. So how does one verify layout differences across these combinations, without going through them manually? Or for instance, diagnosing responsive web issues across multiple mobile and web browsers.
The Resolution: On Demand Environments for Reduced Uncertainty
The next time you are looking to increase automation levels, remember that achieving these shortened delivery cycles are only possible if your test environments is not manual. Having test environments available on demand can be of great help here. TestComplete Environment Manager, which provides an on-demand cloud test lab, is one way of ensuring you can access to a range of browsers, operating systems, and resolutions without any setup and configuration.
Follow the author on twitter @kaulnikhil