What is VM Testing?

Using virtual machines in a virtual testing lab is becoming more popular as a QA testing practice. This approach allows QA teams to save money by running a number of virtual machines on fewer computers and also allows testers to test different application builds on a permanent system configuration. However, the efficiency of testing in virtual labs increases when you automate the testing process with an automated testing tool like TestComplete.

Automated Testing with Virtual Machines

Virtual machines or Virtual PC’s are software like VMware that emulates real computers. By using virtual machines in your QA testing, you can emulate different computers with different OS’s on a single physical computer or create a whole virtual lab, with several differently configured virtual machines. These virtual computers work independently from each other and you can launch two or more virtual platforms simultaneously on one computer, saving the cost of having to purchase more hardware just to run your QA tests. Applications running on a virtual machine behave as if they were running on their own physical system. This is very useful for testing applications and allows testers to test their application under various platforms using a single computer. It can also be helpful for testing web applications since you can simultaneously test your web application in multiple browsers running independently from each other under different virtual machines, again, without the cost of purchasing more hardware for your testing needs.

Testing applications on virtual machines can have different purposes. You can use it for distributed testing of client-server applications, functional testing or in regression testing. But, no matter which kind of QA testing you use, it will be more effective if automated and any kind of testing in virtual labs can be easily automated with TestComplete.

To run your tests on virtual machines using TestComplete, you can buy the Floating User license of TestComplete, activate it on a physical machine and then install TestComplete on virtual machines on your network and use it to run the tests. For more information, see Using TestComplete on Virtual Machines.

Besides virtual machines, you can also use cloud computers for application testing. For more information on running automated tests on cloud computers, see Cloud Testing With TestComplete.

An Affordable, Stable, System Configuration for QA Testing

A computer system is a dynamic system and operating system configurations continuously change. Installing or updating software and hardware drivers happens frequently and installing different versions of an application affects the system’s internal structure, the registry and other crucial components that persuade testing results. While buying multiple computers to support various platforms is an option for some, it is often too expensive for most testing labs.

Since computer systems are ever changing, it causes a problem for QA testing. We cannot fully understand testing results when testing is performed on different configurations. In other words, if our application does not work properly, it is more difficult to determine the reason in a constantly changing system. Another problem is that it is difficult to test different application builds installed on a single operating system. It can cause conflicts and may even provoke system crashes.

Fortunately, virtual machines are a workaround for these problems and are a lot more cost effective. Once you have created and configured a virtual machine or a full lab of virtual machines, you will have a stable system configuration, which is very important when testing applications.

Virtual Testing Labs on a Single Computer

Virtual machines can work independently from each other on a single physical computer, creating a virtual testing lab on one computer. This is a huge benefit which is often used in QA testing.

For example, you can use one computer to test your application under Windows XP, Windows Vista and Windows 7. In virtual labs, there is no need for three different computers to execute tests simultaneously; you can just sequentially execute tests on a single computer with three different virtual configurations. However, you may need a more powerful computer to run several virtual machines on one computer at once, but this is often cheaper than buying three physical computers.

Secure QA Testing in a Virtual Lab With Its Own Network

Sometimes testing applications is an unpredictable process – a tested application may hang the operating system, provoke a system crash and endanger the corporate network’s security. To isolate your testing lab, you can join your virtual labs into a subnet within your network and then isolate it from the rest of the computers in your company. You get a virtual lab network fully isolated from your company’s computers that will increase the security level of your testing process.

Isolating your virtual lab on its own network does not determine the type of software testing you perform; you can still perform distributed testing, GUI testing, web testing and any other kind of testing. You can also remotely launch automated tests on virtual machines and use any automated testing technique provided by TestComplete.

Locking Computers in Your Virtual Lab

QA testing teams may want to execute their automated software tests on locked computers when running tests overnight. Software tests that don’t interact with the GUI can be run on locked computers. GUI tests or functional tests that simulate user actions cannot be run on locked computers, since the user session is frozen (no windows can be shown) and the automated testing tool cannot simulate user actions.

Virtual machine software allows you to run GUI and functional tests on locked computers with TestComplete. To do this:

• Install TestComplete and your application on a virtual machine. Also, copy all files that your application and TestComplete project may require for testing.
• Lock the virtual machine’s host computer.
• Log onto the virtual machine from any other computer.
• Start executing tests.

Note that you can easily automate this process using Automated Build Studio. For more information, see the Automating Tests Using Automated Build Studio section below.

TestComplete will run tests on the virtual machine despite the host being locked. However, if you disconnect from the virtual machine while running GUI tests, the automated tests will fail.

Automating Tests on a Virtual Machine

There are various approaches that let you automate the test runs on the virtual machines. The sections below describe two of them.

Automating Tests Using TestComplete

Usually virtual machine software provides a special COM API that you can use to manage virtual computers. It is easy to work with COM objects from TestComplete, so you can use this COM API to automate testing over virtual machines directly from your virtual testing lab.

The following code sample (VBScript) demonstrates how to automate the running and stopping of a VMware virtual machine directly from an automated test script. We use a .NET wrapper for VIX COM API to power on and then to power off a virtual PC. This script routine works with the VMware Workstation installed on the computer where the routine is launched.

The dotNET object is a special TestComplete object that you can use to call routines and methods that reside in any .NET assembly from your automated tests.

VBScript:

Sub sub1
  Dim vmHost, vmMachine
  Set vmHost = dotNET.Vestris_VMWareLib.VMWareVirtualHost.zctor
  vmHost.ConnectToVMWareWorkstation
  Set vmMachine = vmHost.Open("c:\VMWare virtualmachines\vm1.vmx")

  vmMachine.PowerOn(5000)
  Call vmMachine.Login("tester", "password")
  vmMachine.PowerOff
 End Sub

Automating Tests Using Automated Build Studio

Besides using COM to automate tests from TestComplete, you can also use Automated Build Studio for managing virtual machines. Using Automated Build Studio is the easiest way to automate actions over virtual platforms. It contains a lot of high-level operations intended for working with different VM software. Also, it is not necessary to write scripts or perform other specific tasks. In other words, creating a sample macro does not require a user to have any programming background.

One more advantage is that Automated Build Studio lets you create macros in the visual editor:

Also, it is really easy to specify operation parameters as you do this intuitively via special wizards and dialogs. For instance, to reboot a virtual machine from an Automated Build Studio macro you can just add the appropriate operation to the macro and specify all the required parameters in the enduing dialog. Then, for instance, you can schedule the macro execution to run your virtual machine at a specific time.

So, Automated Build Studio macro can start the virtual machine, launch TestComplete and then close the machine when the test run is over. Below, there is a sample macro created for the purposes described above:

This macro runs the needed virtual machine using the Run Virtual Machine (Virtual PC) operation, waits until the operating system of the machine is loaded completely and then runs TestComplete as a process using the Run Process (WMI) operation.

Note that it is easier to perform some of the tasks mentioned above via TestComplete command line. For example, to launch TestComplete and make it run the desired test automatically, use the /project:project_name /test:test_name command-line argument. You can also make TestComplete to close right after the test execution is over. For this purpose, use the /e command-line key. For more information on running TestComplete tests from a command line, see TestComplete Command Line.

After that, Automated Build Studio checks whether the TestComplete is running. For this purpose, for example, you can check the value of the variable that stores the results of the If Process Is Running (WMI) operation in the While Loop and set a delay each time when this value equals to True.

When TestComplete stops running, Automated Build Studio closes the virtual machine and stops the macro execution.

Automated Build Studio also includes special TestComplete and TestExecute operations for running automated tests with TestComplete or TestExecute from within your Build Studio macros. The operations' properties specify the project, test item or individual script routine to be executed. The operations wait until the testing is over and then save test results to a file or post them to the macro log.

The best practice is to use Automated Build Studio and TestComplete together. This allows you to automate both virtual machines and automated tests in a virtual lab easily.

Running Automated Tests in Virtual Labs with TestComplete

Once you have created automated tests with TestComplete, you should configure the needed virtual machines in your virtual lab. To prepare virtual machines for testing, determine the following requirements:

• Which testing platforms do you need?
• Which VM software should you choose?
• Which automated testing tool do I need to install on virtual machines?

These requirements are important for software testing and you are free to choose any platform and use TestComplete to test your application. You also need to determine how you will work with the tested platforms:

• Will virtual machines run on a single or on several computers?
• Which utility should I use for controlling virtual machines?

For more information on preparing virtual machine for automated testing, see Using TestComplete on Virtual Machines.

Preparing Virtual Labs for Automated Testing

As we mentioned above, you are free to choose almost any testing platform and any VM software available on the market to use in your virtual testing labs.

After you have installed the particular VM software, you can prepare your virtual machines according to the following steps:

1. Create a new virtual machine. For information on how to do this, see the VM software documentation.
2. Configure the virtual hardware.
3. Install the needed operating system.
4. Install TestComplete or TestExecute (see Installing Automated Testing Tool section).

Installing an Automated Testing Tool

There are several tools that can be used to execute automated tests on virtual machines in your virtual lab. For example, you can install TestComplete or TestExecute on the virtual platforms and then run the test on these platforms using the installed tool.

To run your tests on virtual machines using TestComplete, you can buy the Floating User license of TestComplete, activate it on a physical machine and then install TestComplete on a virtual machine on your network. You can then launch TestComplete and execute your automated tests in your virtual lab. Note that all TestComplete plug-ins required for your tests to also be installed and enabled in TestComplete in order for you to use them in your virtual testing lab.

As an alternative, you can create automated tests with TestComplete on a real computer and then copy these tests to several virtual machines in your virtual testing lab. Then you can install TestExecute with all required plug-ins and run your automated tests in your virtual lab. TestExecute is a resource-friendly utility that executes TestComplete tests and logs its results on machines that do not have TestComplete.

For more information on installing TestComplete, refer to the TestComplete Installation Guide document that is supplied along with TestComplete. It describes how you install, modify and uninstall TestComplete in details.

For more information on installing and activating TestComplete license on the virtual machines, see Using TestComplete on Virtual Machines.

Controlling Automated Tests in Virtual Labs

To be able to launch automated tests on virtual machines in your virtual testing lab, they should be running and you should have control over them, whether direct or remote control. There are two possible ways to work with virtual machines in your lab:

• Virtual machines residing on a local computer

  Virtual machines can be installed and run on a local computer, where TestComplete or TestExecute is also installed. You can control different virtual machines just by switching between the windows with virtual computer desktops.

  When running the GUI tests in a virtual machine console, do not minimize the console window. When a virtual machine window is minimized, the operating system does not draw the GUI so TestComplete cannot perform user actions over the tested application.

  As for other kinds of tests, like unit tests, you can minimize the virtual machine console; however, do not shut down the machine until the automated tests are finished.

• Virtual machines residing on a remote computer

  You can access remote virtual machines via a special utility, for instance, the Remote Desktop Connection tool embedded in Windows XP or third-party utilities, VNC, DameWare and others. In this case the virtual machine must be preliminary prepared for controlling it remotely and executing automated tests.

  The Remote Desktop is a built-in Windows feature that lets you access a Windows session running at one computer from another computer. In order to use Remote Desktop, the workstation that has TestComplete or TestExecute must have special server-side software. This software may be included in the operating system (like it is included in Windows XP), or it can be a third-party software like VNC, DameWare or NetOp.

  The Remote Desktop feature lets you perform a variety of tasks. For example, you can use the Remote Desktop if you need to:

  • Observe the tested application’s behavior on remote computers.
  • Run tests on remote computers.
  • Run tests on computers that are locked due to security purposes (see Locking Computers in Your Virtual Lab).
  • View and analyze test results on one or several remote computers, and save them to a local computer.

  To run automated tests with Remote Desktop, follow these steps:

  • If needed, lock the computer where TestComplete (TestExecute) is installed.
  • Log on to the TestComplete (TestExecute) workstation via the Remote Desktop Connection from another computer.
  • Start running tests with TestComplete (TestExecute).

  TestComplete (TestExecute) will run tests in the Remote Desktop session, the same way as if they were running from the remote computer itself.

  When running GUI tests via the Remote Desktop, it is not recommended that you minimize the Remote Desktop window; when the Remote Desktop window is minimized, the operating system does not draw the GUI so TestComplete will not be able to “see” the tested application’s interface and thus will fail to interact with it.

  However, it may inconvenient to always keep the Remote Desktop Window visible as it occupies some part of screen. To solve this problem, you can change the Registry key on the machine from which you access the remote workstation. For more information on this approach, see the TestComplete Tip: Running Tests in Minimized Remote Desktop Windows blog.

Organizing the application’s distributed testing

Also, TestComplete lets you test the application that can work with several clients simultaneously. For example, you can use this approach for testing web sites, client-server or other applications. We call it distributed testing and using it you can run different parts of your test on different machines connected with each other through the network. One more advantage of this technique is that you can easily coordinate and synchronize the test parts running on different machines.

For more information on this testing type, see the Distributed Testing With TestComplete article.

Automated Testing in a Virtual Lab - Scheme

Now that the virtual machine configurations are ready in your virtual testing lab and you know how to control the automated test execution, you can start software testing. The following automated testing scheme allows you to take full advantage of your QA testing in a virtual lab:

1. Create a copy of the virtual machines with the desired system configuration. Only test with this copy from this point forward.
2. Run the virtual machine copy manually or, for example, use the Run Virtual Machine (Virtual PC) operation to automatically do it from Automated Build Studio macro.
3. Install the tested application.
4. Copy all automated tests to the virtual machine in your virtual testing lab.
5. Run automated tests under the virtual machine. As we have mentioned earlier, you can do it manually or you can easily automate it with special operations in your Automated Build Studio macro.
6. Transfer all test results to the physical computer, where they can be analyzed.
7. Turn off the virtual machine copy, for example, you can do it using the Close Virtual Machine (Virtual PC) Automated Build Studio’s operation.
8. Delete the virtual machine copy.
9. Perform steps 1-8 for each automated test in your virtual lab.

With this scheme, you can test different application builds on different operating systems with different configurations.

Determining the Type of Virtual Machine in Automated Tests

TestComplete provides special properties that determine whether TestComplete is running under a virtual machine or a physical computer. With TestComplete, if the Sys.OsInfo.VirtualPC property is True, this indicates that TestComplete is running under a Virtual PC machine. If the Sys.OsInfo.VMWare property is True, TestComplete is running under a VMWare machine.

You can also determine whether TestComplete is running under a remote session via the Sys.OsInfo.RemoteSession property. If this property is True, TestComplete is running under a remote session, for instance, a session launched via the Remote Desktop utility. Using this property you can determine, for instance, whether a tester logged into a virtual workstation via a remote control utility and execute all tests, except GUI tests, on this virtual machine.

You can use any of the properties mentioned above to provide conditional testing in your automated tests. For example, you can execute certain automated tests on specific virtual machines and other automated tests on physical machines, based on particular property values. Also, you can execute only those tests that do not require an Internet connection on virtual machines, and so on.

Distributed Testing in Virtual Labs with TestComplete

Distributed testing means that a test consists of several parts and each part is performed on a separate computer. This testing is often performed with Client-Server type applications. With a virtual lab on one computer, you can perform distributed testing of your Client-Server applications without the need for multiple computers.

With TestComplete, you can automate the whole distributed testing process. Distributed tests can be organized as a master and slave projects in TestComplete, and then run in a virtual lab.

The following image demonstrates how two parts of a distributed test are executed on a single computer. The master project is running on the physical computer and the slave project is running on a virtual machine that runs on that same computer:

Testing Web Applications in a Virtual Lab With TestComplete

Using virtual machines is becoming a good practice for testing different web applications. For instance, you can test web applications simultaneously running in different browsers in a virtual lab. In this case the browsers are completely isolated from each other and this testing approach gives the most reliable results.

The following image demonstrates two instances of a web application running in two browsers – Internet Explorer and Firefox. Internet Explorer is launched on a virtual machine running Windows Vista, Firefox – on a real machine running Windows 7. So, we can perform application testing on machines with different browsers and configurations simultaneously and independently of each other.

Virtual Machine Limitations

There are some obvious benefits of testing applications under virtual platforms; however, they have some limitations too. The virtual operating system can work slower than a real computer, but this depends on the situation: the virtualization technology, hardware, the number of simultaneously running virtual platforms and others. Besides the speed limitations, some hardware functionality can only be used by native operating systems directly working on a physical computer. So, you should take into account these virtual platform limitations, especially if you are testing with specific hardware features, multiprocessor systems and other important factors. Of course, these limitations depend on the particular VM software and hardware and you should check their documentation first before performing automated testing.

There is a lot of VM software available on the market. TestComplete can work with any virtual machine software, including the most popular for which Automated Build Studio includes a special support:

• Microsoft Virtual PC 2004 and 2007 (Microsoft Virtual PC operations category)
• Microsoft Virtual Server 2005 (Microsoft Virtual Server operations category)
• Microsoft Hyper-V for Microsoft Windows Server 2008, Windows Server 2008 R2, Windows Server 2012 and Windows 8. (Microsoft Hyper-V operations category)
• VMware Server 1.0 or later (VMware Server operations category)
• VMware Workstation 5.5 or later (VMware Workstation operations category)
• VMware vSphere 5 or later (VMware vSphere operations category)

No matter which VM software you use, you can easily automate your QA testing in a virtual lab with TestComplete.

Conclusion

Automated testing in virtual labs and working with virtual machines allows QA testers to get the maximum benefit out of their testing with the lowest overhead cost. TestComplete’s ability to work with both virtual machines and real computers gives testers an edge and can even increase your testing coverage and efficiency. TestComplete also helps you achieve your maximum ROI when using automated testing in virtual labs. If you haven’t tried TestComplete, be sure to download and try it today.

If you are interested in automating the entire process of running the tests on virtual machines, be sure also to install Automated Build Studio on your local machine.

If you want to learn how to use cloud computers in your testing, how to prepare them and automate the testing process, see the Cloud Computers With TestComplete and Cloud Test Automation Tutorial articles on our web site.