Automate Detox Mobile Tests on Real Android Devices in the Cloud

  April 11, 2018

Every now and then a new test automation framework emerges, it quickly gains attention from developers and testers. This is happening right now with the Detox framework for automating mobile tests.

A New Framework

Detox is a JavaScript mobile testing framework that is built into the application and the test execution starts with app launch. This makes test execution really fast and robust as no external additional tools are needed to orchestrate and synchronize during the test execution.

As for the popularity of Detox, it can be easily summarized.

  • First, it comes to the number of users. There are plenty of seasoned JavaScript web developers using libraries such as ReactJS, while experienced mobile programmers are scarce.
  • Some existing testing frameworks that looked still really promising a year ago are now fading out and users are forced to move to other frameworks. This also happened to Calabash which is no longer under development.
  • The current framework leader Appium originated from the web app testing and was later widely utilized in mobile app testing. It was great for testers as one could use the same concept in web testing with Selenium and in mobile testing against real devices or simulators. The problem with Appium is that launching tests is rather slow with installing and starting communication with the Webdriver.

Running Detox in Real Device Cloud

Bitbar Cloud has always been about openness, providing great freedom for end-users to choose how to test their application. This applies also to running Detox tests on Bitbar Device Cloud.

Follow these steps below to start an Android real device Detox test run on Bitbar Cloud. Before that, let’s check how your ‘ run-tests.sh ‘ should look like.

  1. You should already have locally a ‘ package.json ‘ file listing your needed npm packages to use with your Detox test. You will need to include this file in your test package zip and upload it to Bitbar Cloud.
  2. Install all npm packages required to run your Detox tests: ‘ npm install ‘. This uses your package.json file to select all the required packages.
  3. Start Detox server: ‘ "${PWD}/node_modules/.bin/detox" run-server > detox-server.log 2>&1 &
  4. Connect device to server: ‘ adb reverse tcp:8099 tcp:8099 ‘, this same port number should be declared in package.json file: ‘ "server": "ws://localhost:8099", ‘
  5. Package.json file needs to contain the device id of the device to run test on, use this command to set the correct device id: ‘ sed -i.bu "s/ADD_DEVICE_ID_HERE/$UDID/" package.json
  6. Your script is now ready to start the app and test execution! ‘ "${PWD}/node_modules/.bin/detox" test –configuration android.device.release –loglevel verbose > detox.log 2>&1

Once everything is ready, we can move to Bitbar Cloud for test creation.

start an appium android server side test
  1. On Bitbar, create a new Appium Android Server Side project and name it as, for example, “Detox Android”.
  2. Start a new test run:
    • Upload a small dummy app. The testing framework requires you to upload some apps, even if this is not the app that you’ll be testing. The app to be tested is uploaded to the test suite zip.
    • The test zip package needs to include everything required for running your test. This includes your app with built-in Detox tests and your ‘ run-tests.sh ‘ that installs Node dependencies and launches the app under test.
  3. Select Android devices to test on, note that Detox can run on Android devices OS version 5.0 and up.
  4. Start your test run!
upload android test files

Once your test is ready, you can find all the results in Bitbar Cloud, screenshots, video recording, logs and device performance to compare execution between devices.

detox tests console

Test results for Detox tests on Bitbar Cloud

Here’s a working sample if you want to get your hands dirty: https://github.com/jarnie/testdroid-samples/tree/master/detox/sample-react-native-bitbar

Final Thoughts

Test automation has historically been thought through a manual testers’ point of view. It should be possible to interact with the application as it would be used by a real tester.

This is the reason for the success of current mainstream frameworks (eg. Appium). But is this really a valid point still today with DevOps and the “shift left” concepts? With developers ever more engaged with creating and maintaining test suites, why not make use of the internal workings of the application when testing it?

This is what Detox is doing, moving away from black-box testing to gray box testing, where developers make use of the application’s internal state to drive tests. If you have Detox tests already and are in need of a mobile device cloud, you can already try to automate the tests through our free trial here.