Robot Framework is a generic keyword-driven test automation framework for acceptance level testing and acceptance test-driven development (ATDD).
Many of our Bitbar users have been using this framework for a basic acceptance testing that extends the system level testing capabilities with specifications and test cases associated with the actual app testing.
However, if you are not familiar with Robot Framework or if you look for a way how to use it for robust mobile app testing on Bitbar, I’ll walk you through some basic things in this blog.
What is Robot Framework and How Does It Work
Robot Framework is extremely easy to set up, use and modify to get both Android and iOS apps tested.
The test syntax that it uses is based on keywords and these keywords are quick to edit and further configure to make a match with the application under test.
Furthermore, testing capabilities provided by Robot Framework can be easily extended with the test libraries which can be implemented using Python, Java and even some other languages are supported.
The nifty thing with Robot Framework is how those mentioned keywords work. Users can quickly create new keywords, either using the existing example ones or by writing everything from scratch.
If you look for more generic information about Robot Framework, there are plenty of great examples and online documentation on GitHub.
The Basic Setup
To get started with Robot Framework basically you need Python and pip installed. If you have the environment properly configured you can just execute the installation command lines:
pip install robotframework
pip install robotframework-appiumlibrary
After you have installed Robot Framework, you can take a look at the example where we use our Bitbar Sample Application with very basic Robot Framework test examples.
For the cloud run, you basically need a Python script and shell script that makes sure the test execution on the cloud end works fine. For instance, you need to add the pip installation in your shell script to ensure everything gets properly installed on your cloud session as well.
In this example, we’re using Appium as a basis to run the test and make sure everything gets logged for further inspection.
## Cloud setup
echo "Starting Appium ..."
/opt/appium/appium/bin/appium.js --log-no-colors --log-timestamp --command-timeout 120 >appium.log 2>&1 &
# Wait for Appium to fully launch
sleep 10
ps -ef|grep appium
echo "Extracting tests.zip..."
unzip tests.zip
echo "Installing pip"
curl -O https://bootstrap.pypa.io/get-pip.py
python get-pip.py --user
echo "Installing requirements"
~/.local/bin/pip install -r ./resources/requirements.txt --target .
## Start test execution
echo "Running test"
pybot -x TEST-all tests/android_example.robot
This should be all that you need for a shell script. To create a compatible test you need to add a few lines to ensure all paths are recognized and test cases can be found from the right fileset. The file structure for the test package can be something as follows:
The following example includes both Android and iOS applications in the test package (under resources ->app folder). The test definition can be found under ‘tests’ and testing libraries are included under libs->examplelib. In addition, you can easily and quickly tweak and create your own test libraries with the instructions shown on the Github documentation of Robot Framework.
What Data Can Robot Framework Produce
When running a Robot Framework test in Bitbar Cloud users do not need to do any significant tweaks for their test runs (or job configs). The test automation and execution works the same way as with any other framework, and results are quickly available from any of those Android and iOS devices.
If you are starting out test automation withBitbar Cloud and planning to use Robot Framework, simply create a project (Android or iOS), upload your application (APK or IPA), upload the test package, and select whatever devices you want to use for a test run. Just in a few minutes the test run will be done (naturally this depends on the length of your test case) and the following data can be fetched from a test run:
This example includes three basic test cases (we’ll come back to those in the next chapter of this blog). However, after the test run is finalized you have now all the data in one, compact view of Bitbar Cloud. If you click any of those files shown in the overview you’ll get full log opened in the view below:
For instance, the shell script file that we introduced in the first chapter of this blog extracted the tests.zip, installed the PIP, setup other tools and finally executed the example tests. The output of the log can be also inspected in this view – and for example, it will look something like this:
Imports done
==============================================================================
Tests
==============================================================================
Tests.Example
==============================================================================
Simple Smoke Test - Correct Answer | PASS |
------------------------------------------------------------------------------
Simple Smoke Test - Wrong Answer | PASS |
------------------------------------------------------------------------------
Simple Smoke Test - Wrong Answer 2 | PASS |
------------------------------------------------------------------------------
Tests.Example | PASS |
3 critical tests, 3 passed, 0 failed
3 tests total, 3 passed, 0 failed
==============================================================================
Tests | PASS |
3 critical tests, 3 passed, 0 failed
3 tests total, 3 passed, 0 failed
==============================================================================
Output: /home/ubuntu/test/output.xml
XUnit: /home/ubuntu/test/TEST-all.xml
Log: /home/ubuntu/test/log.html
Report: /home/ubuntu/test/report.html
In addition, to get a comprehensive understanding of how the app (with the test) really does on the physical device, you’ll also get performance data about the CPU and memory consumption:
Okay, that’s some fundamentals for creating test runs using Robot Framework and managing the data after tests are done. Let’s take a look at how easy those test cases are to build and what you need to know about Robot Framework to use it efficiently with Bitbar Testing.
How to Create Robot Framework Compatible Test Cases
First, some generic settings are defined for the application and test session. Resource links to a file where all common procedures are provided in the test package. Test Teardown provides instruction of what system should do when the test session is done:
*** Settings ***
Resource ...${/}resources${/}common.robot
Test Teardown Close Application
For the variables, creator of a test should define some basic things here. In the Bitbar sample application, there is a name field (editText) and one button (button1). IDs and actions can be defined in this section:
*** Variables ***
${NAME} John Doe
${INPUT_NAME_FIELD} id=com.bitbar.testdroid:id/editText1
${SUBMIT_BUTTON} id=com.bitbar.testdroid:id/button1
The example application looks like this:
Now, let’s look at those test cases (using Android). To create three different test cases for the application, we start with a simple smoke test where the correctness of selection is validated. The correct answer in this scenario is when the option to use Bitbar Testing is clicked and the name is entered in the text field. Two other test cases indicate the wrong answer with radio buttons (Option to buy 101 devices and ask mom to help you). Test cases are as simple as follows:
*** Test cases ***
Simple Smoke Test - Correct Answer
[Tags] cloud
Set Up And Open Android Application
Input Name ${NAME}
Select Option Use Testdroid Cloud
Submit Selection
Validate Correct Answer
Simple Smoke Test - Wrong Answer
[Tags] 101
Set Up And Open Android Application
Input Name ${NAME}
Select Option Buy 101 devices
Submit Selection
Validate Wrong Answer
Simple Smoke Test - Wrong Answer 2
[Tags] mom
Set Up And Open Android Application
Input Name ${NAME}
Select Option Ask mom for help
Submit Selection
Validate Wrong Answer
When test execution is started, Robot Framework first parses the test data. After that, it uses keywords provided with the test library/libraries to interact with the system.
In the keyword example, there is an input name, select of options (3 radio buttons) and submit a selection (=a button press). In addition, the keywords section can include validation of correctness. In the example below, selecting the right radio button uses XPath to find a name included in the actual UI element:
*** Keywords ***
Input Name
[Arguments] ${name}
Input Text ${INPUT_NAME_FIELD} ${name}
Hide Keyboard
Select Option
[Arguments] ${option}
Click Element xpath = //*[contains(@text, '${option}')]
Submit Selection
Click Element ${SUBMIT_BUTTON}
Validate Correct Answer
Wait Until Page Contains Congratulations ${NAME} 5s
Validate Wrong Answer
Wait Until Page Contains Wrong Answer 5s
Conclusion and Download Link
Robot Framework is very handy and easy to use. Some of you have been using it in the past so in case you have your tests implemented, and you have your Android / iOS app ready for the test, give this framework a try on Bitbar Cloud. You’ll be surprised at how smoothly things work and how quickly you can do acceptance testing. Furthermore, here is the full test package available for a download (and tweaks).
Don’t forget to try out your Robot Framework tests at Bitbar Cloud for free!