Appium is one of those frameworks that scale well across different use cases. For example, many of today’s popular mobile games do get tested using Appium and Image Recognition. This is an extremely easy and effortless way to create tests for graphics-based content and mobile games are natural to use cases for Appium Image Recognition.
We’ll take a look at Appium image recognition features, how to get started with this setup and use real Android and iOS devices from Bitbar Testing.
Why Using Image Recognition with Appium?
Image recognition technique (sometimes also called OCR, Object Character Recognition) has become popular among mobile game developers as they find it very easy to use and handy for creating automated tests for their mobile games. It’s worth mentioning that image recognition use cases are not limited to mobile games but image recognition can be used together with Appium also for native mobile apps, both on Android and iOS. To successfully use Appium for mobile game testing, download and follow our free Appium beginner’s guide to build up your Appium environment.
The main thing with Appium image recognition is that tests are easy to create and those tests would work on both platforms (cross-platform usability). The need for using image recognition for mobile games is coming from the UI components. When graphics content (from game engines, strict OpenGL ES content) is used on screen, many of these test automation frameworks cannot recognize UI elements characteristics, such as IDs, element attributes, other UI characteristics and so on. Despite human recognizes a button or other visual element on the screen, test automation scripts sees that only as pixels.
Because of this, image recognition has become an awesome way to automate mobile game tests simultaneously even on hundreds of real mobile devices.
Required Components for Appium Image Recognition
The following components and software are required to get started with Appium image recognition:
- Bitbar Testing – a cloud-based app testing service for automating tests on real mobile devices.
- OpenCV – OpenCV (Open Source Computer Vision Library) is an open source computer vision and machine learning software library. OpenCV was built to provide a common infrastructure for computer vision applications and to accelerate the use of machine perception in the commercial products. Being a BSD-licensed product, OpenCV makes it easy for businesses to utilize and modify the code.
- Bitbar Akaze – fork of official Accelerated KAZE repository with JSON support added.
- jsoncpp – jsoncpp provides C++ support for example. With the help of jsoncpp, it can be used to represent numbers, strings, ordered sequences of values, and collections of name/value pairs. This is added to to the Bitbar Akaze library. NOTE! If you are looking to build Akaze and OpenCV for other platforms, you basically need this one.
- Maven – Used to compile the test code and launch it.
Using these components with Appium and Bitbar Public Cloud you are able to get your test runs started and execute in a timely manner.
Example – Using Java for Appium Image Recognition
First of all, take a look at our Appium image recognition example at Github. This provides an out-of-the-box example for Java using OpenCV and Bitbar AKAZE algorithm to execute Appium test scripts.
The example project can be used for local tests or with Bitbar Public Cloud and Private Cloud both for client-side and server-side execution. The only difference between using the client-side and server-side execution from the script point of view is the Appium Desired Capabilities and the need to tweak those to work against the back-end (Bitbar Public Cloud and real devices).
Installing and Setting Up OpenCV
The OpenCV java libs can be found under ./lib//opencv/ directory inside the project. In order to install libraries locally with maven, just simply execute the following command line:
mvn install:install-file -Dfile=lib//opencv/opencv-2413.jar -DgroupId=opencv -DartifactId=opencv -Dversion=2.4.13 -Dpackaging=jar
Note that each platform has its own version of the libraries, so modify the command accordingly. If a different (newer) version of OpenCV is used, OpenCV version number needs also to be updated in pom.xml file.
Installing and Setting Up Bitbar Akaze fork
The AKAZE C++ implementation can optionally be found and built from Bitbar Akaze fork. The current project contains the ./lib//akaze/ folder with pre-built binaries for Linux, Mac and Windows. The sample project uses only the akaze_match binary currently.
Everything else is fetched by Maven automatically.
Additional Information on OpenCV and Bitbar AKAZE Fork
The example introduced above relies on AKAZE algorithm to find the matching key points from two images and store them into a JSON file. Furthermore, the example uses the OpenCV Java bindings to process the JSON file with key points and find the homography of the wanted image in a scene (a screenshot from device screen).
The AkazeImageFinder.java class will run akaze_match, then use the key points from JSON to try to identify a given image inside a scene and return the corners of the image found.
In this example, we look for specific images from ./queryimages/ folder and try to find them on a screenshot.
Once the image is found, the test script can tap/click it, drag and otherwise interact with the game, do more holistic gestures (swipes, zooms, pinchs etc.) for the game running on the device. Full reference to all possible interactions and UI commands can be found in the TestdroidImageRecognition.java class.
The class has different functions for working with and manipulating the image captured on the screen and also finding those images. These are then used inside the test classes for the game tests. Few examples of used functions as follows:
// returns the corners of the image if found, or null if no image was found
public Point[] findImage(String image, String scene)
// takes a screenshot, then tries to find the image on it and return the corners
public Point[] findImageOnScreen(String image)
// takes a screenshot, tries to find the image on the screen, then tap the middle of it
public void tapImageOnScreen(String image)
// takes a screenshot, finds the image on the screen, taps and holds it, then swipes the screen while still holding it
public void swipeScreenWithImage(String image, int repeats)
// takes a screenshot, tries to find the image, taps it and drags it to a certain position given as x and y offsets from middle of the screen
public void dragImage(String image, double x_offset, double y_offset)
How to Run Appium Tests locally using CommandLine with Maven
This sample project uses the Bitbar Sample App available in this repository at apps/builds:
The project expects to find these applications (APK for Android and IPA for iOS) at the root of the project with names application.apk and application.ipa. You’ll also need to have an Appium server launched. For iOS, the UDID of the device should be preset in the server settings – if you use a local environment and devices. For test runs on Bitbar Testing you basically don’t need to preset or configure UDIDs or sign the app with a certain provisioning profile as it is taken care of for you automatically when the test is initiated.
On your local environment, tests can be executed with Maven using the following command:
mvn -Dtest= test
For example, to execute only the test “mainPageTest” from the class AndroidSample.java you can use the following command:
mvn -Dtest=AndroidSample#mainPageTest test
To first clean all the previous test result files, add keyword clean to the command:
mvn -Dtest=AndroidSample#mainPageTest clean test
You can also use an IDE to launch the tests. Make sure the project is correctly imported as a Maven project and that the pom.xml file has been correctly discovered for dependency management.
Running Tests on Bitbar Testing as Server-Side Run
First, you need to create a project for Appium test run on Bitbar Public Cloud. This can be done by manually logging in and creating a project OR alternatively you can use a Bitbar API to create your project. With cURL it would be created this easily:
$ curl -X POST -d "name=NewProject" -H "Authorization: Bearer abcdefgh-1234-ijkl-m5n6-opqrstuvxyxz" https://cloud.bitbar.com/api/v2/me/projects
More information about Bitbar API can be found in this cheat sheet.
Another thing you to configure before starting your test run is the project type how you want your test run to be executed. It can be Appium Server Side for Android or iOS depending on which devices you want tests to run.
Once you have your project created, use the provided scripts to create the test zip from your project:
./createAndroidZip.sh
// or alternatively ...
./createiOSZip.sh
Now that you have your test zip and application file (the example uses the Bitbar Sample App available in this repository at apps/builds), you’ll be ready to begin creating test runs in your project.
First, upload the app and test zip in Bitbar Public Cloud, choose the device group and finally make sure you have set high enough timeout for your test in the Advanced options (default is 10 minutes).
If you change the name of your Android or iOS test class, you will need to update it to the run-tests_android.sh and run-tests_ios.sh TEST and JUNIT variables as appropriate:
# Name of the desired test suite and optionally specific test case, eg: AndroidSample#mainPageTest
TEST=${TEST:="AndroidSample#mainPageTest"}
# JUnit file wont have the #caseName ending
JUNIT="AndroidSample"
OPTIONAL: Building the Latest Libraries for Image Recognition
This example has been tested on Mac Yosemite. The very same steps also work on major Linux distros.
First, we’ll need to compile up to date version of jsoncpp (needed by Akaze later on). At least on Mac, jsoncpp installed through brew gave linking errors while linking with Bitbar Akaze.
jsoncpp
Python 2.6 is needed to extract amalgamated source and header files:
git clone https://github.com/open-source-parsers/jsoncpp.git
cd jsoncpp
python amalgamate.py
By default, the following files are generated and are needed to compile new AKAZE binaries.
- dist/jsoncpp.cpp: source file that needs to be added to your project.
- dist/json/json.h: corresponding header file for use in your project. It is equivalent to including json/json.h in non-amalgamated source. This header only depends on standard headers.
- dist/json/json-forwards.h: header that provides forward declaration of all JsonCpp types.
You will also need to build and copy the libjsoncpp.a library to your local lib directory:
mkdir build
cd build
cmake ..
make
cp src/lib_json/libjsoncpp.a /usr/local/lib/
OpenCV
Start with the following command to build a new library for OpenCV:
git clone https://github.com/Itseez/opencv.git
cd opencv
git checkout 2.4
mkdir build
cd build/
cmake -G "Unix Makefiles" ..
make -j8
make install
This creates Java jar files for image recognition sample. Latest jar files are installed by executing the following command:
mvn install:install-file -Dfile=path/to/opencv/build/bin/opencv-2413.jar -DgroupId=opencv -DartifactId=opencv -Dversion=2.4.13 -Dpackaging=jar
The pom.xml file needs to be updated with installed OpenCV version (2.4.13 in this case).
Bitbar AKAZE
You can build Bitbar’s AKAZE fork using the following commands:
git clone https://github.com/bitbar/akaze.git
cd akaze
# copy previously built versions of jsoncpp to 'src/lib/jsoncpp/', 'src/lib/json/' and src/lib/jsoncpp.cpp
mkdir ./src/lib/jsoncpp/
cp -r path/to/jsoncpp/dist/json ./src/lib/jsoncpp/
cp -r path/to/jsoncpp/dist/jsoncpp.cpp ./src/lib/
mkdir build
cd build
cmake ..
make
cp bin/akaze_* path/to/image-recognition/akaze/bin/
That’s all folks! You should be now up and running with the latest assets for Appium image recognition and running your tests smoothly on the environment of your choice!
Happy Mobile Game Testing!