Appium Tip #23: Parallel Test Runs on Android and iOS
Emulators and simulators have their own pros and cons when used as a part of the application development. Using emulator is common in the earliest phase of the development and the need for using the real device typically comes up when the application reaches the first usable version with enough logic and implementation in it. However, this is a logical path for app developers as emulators and simulators have severe limitations, not only in how they handle real hardware details but also how many concurrent instances can be launched simultaneously. Real devices do not have this problem and any number of available devices can be used for development and testing purposes. In this blog of the series, we’ll talk about Appium parallel test automation on Android and iOS real devices.
Unfortunately, Android emulator has lots of issues with its parallel test run execution. If you try to run several device emulations simultaneously, you’ll soon face issues with memory running out and emulation instances taking too much of your memory and CPU from the development environment itself. It’ll get very challenging (and eventually slow) to run anything with a meaningful number of different ‘devices’ in a simulated environment.
With iOS simulators and Xcode environment, it’s not even possible to run local parallel test runs on the same machine as iOS simulator cannot be launched multiple times. And those are yet simulators, providing no real hardware, infrastructure, software and all dependencies that are used in the real user-device context.
Appium Parallelism and Concurrency
Test automation is meant to enable parallelism and concurrency. Running a test script on any number of devices should be always the goal when using test automation as it enables fast test run launches and instant results, and the possibility to quickly iterate issues on the app under development. In addition, manually starting test runs device by device doesn’t fit well to this sort of agile thinking with mobile test automation.
Because of this sort of limitations test automation using Appium should be always unlimited to any concurrent test runs, on any number of real mobile devices. That makes the development and testing agile with faster turnaround, and result in better quality and more robust apps. The major difference is how the test is launched, not in how it is executed or what number of devices are selected for testing. The picture below illustrates in its simplicity how these two things are different:
A quick glance gives the impression that there isn’t much difference between these two. First one (above) illustrates an Appium session where each mobile device is initiated separately, the test session will be conducted normally on those devices, and eventually, results are pulled back to the environment/computer from where the test was launched. The below illustrates a use case where test and application have been uploaded to the server that automatically starts sessions on selected (not hardcoded) devices, and maintains results in the same location. Let’s drill deeper with these examples.
Appium desired capabilities take various aspects into consideration: automation engine, full platform definition (OS, version etc), device to be used and its parameters (name, orientation, locale etc), application (name/link, run details, timeouts), and many others related to launch, execution or device session. For instance, setting (hardcoding) something in desired capabilities wouldn’t make it easy to scale up for a myriad of devices, but instead, all these would need to be hardcoded in these desired capabilities. The nature of this comes from the Selenium and Webdriver where no additional devices were meant to be used for the execution and it only took a scale of different browsers etc. into consideration.
Again, due to these sort of limitations and maximize the potential of Appium the test execution should be used in the context where all possible preset and hardcoded details are minimized and test script can be easily used across devices, for sake of concurrency and any number of parallel test runs.
Starting Appium Tests in Consecutive Order
Appium tests can be easily started from the localhost on real mobile devices using cloud mobile app testing service. If you start your test runs from localhost you have to configure all devices that app will be installed on and tests will run against the app in parallel mode. This is called the client-side execution where tests are launched from the localhost, the application is uploaded separately on a server that then automatically takes care of installing the application to a dedicated device. The device is selected in the desired capability and must be defined before the test is launched:
capabilities.setCapability("testdroid_device", "Samsung Galaxy S7 SM-G930F");
After you’ve started this test, you need to start another one with a different device. Easiest implementation is naturally to build an array of devices in your test script and make sure tests launched against each and every device listed on that table. The instigator script and table could also use DeviceFinder to go through all possible devices and select the first free one.
The following picture illustrates the logic behind client-side execution. All details for the test – setting up a webdriver, desired capabilities and test itself – is done on a local machine. It communicated with Appium Broker that handles all test steps and makes sure the dedicated device is properly in use for the test session. In between there is an Appium Server running on a server infrastructure that does the actual communication, passing the test, data and results between the server and the device.
As said, it’s possible to get a decent level of concurrency with client-side execution, but the test script must be prepared so that each device is reserved, accessed and test will run on it separately. Also, fetching test results (data, logcats, Appium log, screenshots etc) must be done separately for each device and test run.
Starting Appium Tests in Parallel
When it comes to true parallelism with Appium you should strive to start tests at the same time. This is enabled with server-side execution where application and tests are automatically uploaded to the server from CI/CD or with some other method via API. There is a significant difference compared to client-side execution as everything will now reside on the server that communicates, runs and controls the Appium sessions on real physical devices. As the test can pretty much identical for both Android and iOS, however, devices cannot be mixed in the test run. UI elements, the platform and many others work differently on Android and iOS so the user must select whether test run will be done on either one of these platforms.
The server-side execution brings in the true parallelism and scales up depending on test execution needs. When the cloud service is implemented correctly, there are no bottlenecks or things slowing down the parallel execution of test runs. This is why Bitbar Testing provided an unlimited number of devices for server-side Appium test execution.
One of the benefits also to use server-side execution is that you don’t need to configure any test run specific details in desired capabilities, but only those for application, package and how you want it to get executed. Parallelism in this context also means that the test package will be identical on each device and even on iOS the system will take care of provisioning packages. In few words, this is the feature that makes Appium run simultaneously on yet hundreds of unique devices. Sign up for a free trial of Bitbar Testing and try Appium parallel test runs!