Testing Flutter Apps on Real Android Devices with Flutter Driver

Jarno Niemela

December 06, 2019

Since its initial release, Flutter has quickly gained its popularity among developers for building beautiful Android and iOS applications. Like apps built with any other development toolkit, automated testing of Flutter apps is the only way to ensure app quality in the shortest time possible.

In this article, I’d like to talk about how to create the unit, widget and integration tests for automating the testing of Flutter apps and execute them against real Android devices in Bitbar Cloud.

Creating a Sample Bitbar App with Flutter SDK

To better understand how to automate Flutter app testing, I started creating a Bitbar sample app using Flutter SDK (see UI below).

The MainPage looks like this:

Text element
3 button elements (RaisedButton)
TextField element
Image asset element

The SubPage looks like this:

2 Text elements
button element (RaisedButton)
TextField element
Image asset element

In my opinion, the easiest way to create a new Flutter app is to use the flutter create command, for example: flutter create my_app. This will create a sample app for Android and iOS.

I created the sample app by modifying this sample app. The app source is in a file called main.dart, and it is in the lib directory.

Note: I gave all the important UI elements Key values, for example:

key: Key('question-text')

Creating Unit and Widget Tests

A ‘unit test‘ is to test a single method or class and a ‘widget test‘ is to test a single widget. Here, a ‘widget‘ means UI elements like layout, button, text box, etc.

Unit tests require a test package (https://pub.dev/packages/test), and the flutter_test package provides additional tools for widget testing.

1. Add the test or flutter_test dependency

You can use the following approach to include test or flutter_test (or both) dependency on the app’s pubspec.yaml file:

dev_dependencies:
  flutter_test:
    sdk: flutter
  test: any

2. Create a unit test file

Create a test directory and test file inside that directory. It is also a good idea to make separate directories for unit and widget tests.

In this example, we can create a file ending with _test.dart for example main_test.dart.

Import packages:

import 'package:test/test.dart';
import 'package:my_app/main.dart';

Sample test:

test('correct answer', () {
  final mainPage = MainPage();

  changeText(true);
  expect(subPageAnswerText, correctAnswerText);
});

3. Create a widget test file

Again create a file ending with _test.dart.

Import packages:

import 'package:flutter/material.dart';
import 'package:flutter_test/flutter_test.dart';
import 'package:my_app/main.dart';

Sample test:

testWidgets('correct answer test', (WidgetTester tester) async {
  final app = App();
  await tester.pumpWidget(app);
  expect(find.text("What is the best way to test your applications against over one hundred devices?"), findsOneWidget);
  await tester.tap(find.byKey(Key('correct-answer')));
  await tester.pumpAndSettle();
  expect(find.text("You are right!!!"), findsOneWidget);
});

4. Use command to run the tests

Unit tests can be run with the command:

flutter test test/unit/main_test.dart

And widget tests can be executed with the command:

flutter test test/widget/main_test.dart

5. JUnit Report

One thing to note: If the unit and widget tests are executed in Bitbar Cloud within a CI tool e.g. Jenkins, the test results will not display correctly after the tests are finished.

To get over this, we can use a Flutter package to convert the test results to the JUnit XML report format. (https://pub.dev/packages/junitreport).

First, make sure the following stuff is included in the system path:

flutter/.pub-cache/bin
flutter/bin/cache/dart-sdk/bin

Then, install the junitreport package by running the command below:

flutter pub global activate junitreport

Both unit and widget tests can be run with this command:

flutter test --machine | tojunit > TEST-all.xml

After the tests are finished, the unit and widget test results can be found in a file called TEST-all.xml.

Creating Integration Tests

An integration test tests the complete app and is isolated from the app under test. Integration tests require a flutter_driver package (https://api.flutter.dev/flutter/flutter_driver/flutter_driver-library.html).

Flutter driver:

The application runs in a separate process from the test itself
Android (Espresso)
iOS (Earl Grey)
Web (Selenium WebDriver)

1. Add the flutter_driver dependency

The flutter_driver package needs to be added to dev_dependencies section of the app’s pubspec.yaml file:

dev_dependencies:
  flutter_driver:
    sdk: flutter

2. Create an integration test file

Create a directory called test_driver. Add files main.dart and main_test.dart (or something ending with _test) inside that directory.

Main.dart

This first contains an ‘instrumented’ version of the app being tested. It enables Flutter driver extensions and runs the app.

import 'package:flutter_driver/driver_extension.dart';
import 'package:my_app/main.dart' as app;
void main() {
  enableFlutterDriverExtension();
  app.main();
}

Main_test.dart

This file is created to contain the test suite.

Import Flutter Driver API:

import 'package:flutter_driver/flutter_driver.dart';
import 'package:test/test.dart';

Connect to the app in “setUpAll” method:

setUpAll(() async {
  driver = await FlutterDriver.connect();
});

Close connection in “tearDownAll” method:

tearDownAll(() async {
  if (driver != null) {
    driver.close();
}
});

3. Write a sample integration test:

test('correct answer', () async {

  final goBackButtonFinder = find.byValueKey('back-button');
  final correctAnswerButton1Finder = find.byValueKey('correct-answer');
  final answerTextFinder = find.byValueKey('answer-text');
  String correctAnswerText = "You are right!!!";

  Health health = await driver.checkHealth();
  print(health.status);

  SerializableFinder goBackButton = find.text('Go back');
  try {
    await driver.waitFor(goBackButton);
    await driver.tap(goBackButtonFinder);
  } catch (e) {
    print('try again not visible');
  }

  await driver.tap(correctAnswerButton1Finder);
  expect(await driver.getText(answerTextFinder), correctAnswerText);
});

4. Use command to run the integration tests

Now that we have an instrumented app and a test suite, we can run integration tests with the following command:

flutter driver --target test_driver/main_test.dart

There doesn’t seem to be a ready package for converting integration test results in a format that could be read in a CI tool, at least at the time of writing this. Test results must be parsed and converted to Junit XML format or something else.

5. Take screenshots

Screenshots can be taken in the integration tests with the ‘screenshots’ package (https://pub.dev/packages/screenshots).

Add the dependency in the pubspec.yaml file (the current version of the package at the time of writing this blog):

dev_dependencies:
  screenshots: 2.1.1

Create the screenshots.yaml file inside the project root directory. It should look something like this:

tests:
# Note: flutter driver expects a pair of files eg, main1.dart and main1_test.dart
  - test_driver/main.dart

# Interim location of screenshots from tests
staging: /tmp/screenshots

# A list of locales supported by the app
locales:
  - en-US

devices:
  ios:
    iPad 4 A1459 10.2:
      frame: false
  android:
    Samsung:
      frame: false

# Frame screenshots
frame: false

Add these in the main_test.dart file:

Import dependency:

import 'package:screenshots/screenshots.dart';

Create a config:

final config = Config();

Take screenshot inside test method like this:

await screenshot(driver, config, 'correct-answer');

Testing Flutter Android Apps on Real Devices in Bitbar Cloud

In Bitbar Real Device Cloud, Flutter tests are executed under the Appium Server Side type test project. The ‘run-tests.sh’ shell script is used to run the Flutter tests. Integration, Unit and Widget tests can be run in the same project in the cloud.

Note that only integration tests actually install and run the test in a device, unit and widget tests don’t require devices to run. Device time is still spent when running unit or widget tests, the device is just idling while the test is running.

See a quick demo below.

Unit and Widget tests

Install JUnit report:

flutter pub global activate junitreport
tojunit --help

Run unit and widget tests:

flutter test --machine | tojunit > TEST-all.xml

Move test results to root directory so that they can be found by Jenkins:

cd ..
mv my_app/TEST-all.xml TEST-all.xml

Below is the content of the ‘run-tests.sh’ file

#!/bin/bash

# run-tests file for flutter unit and widget tests
flutter --version
# see what Android SDK is installed
echo $ANDROID_HOME
android list target

echo "Extracting tests.zip..."
unzip tests.zip

# add flutter to path
FLUTTER_PATH="/opt/testdroid/flutter/bin"
PUB_CACHE_BIN="/opt/testdroid/flutter/.pub-cache/bin"
DART_PATH="/opt/testdroid/flutter/bin/cache/dart-sdk/bin"
export PATH=$PATH:$FLUTTER_PATH:$PUB_CACHE_BIN:$DART_PATH

# install Android SDK platform 28 (this can be removed when cloud VMs contains correct version)
echo "install android SDK"
wget https://dl.google.com/android/repository/sdk-tools-linux-4333796.zip
unzip sdk-tools-linux-4333796.zip
# rename old tools and move the new one to its place
mv $ANDROID_HOME/tools $ANDROID_HOME/tools-old
mv tools $ANDROID_HOME/
yes | /root/android-sdk-linux/tools/bin/sdkmanager "platforms;android-28" "build-tools;28.0.3" "platform-tools"

echo "run flutter doctor"
flutter doctor
cd my_app

echo "install junit report"
flutter pub global activate junitreport
tojunit --help

# (build app) get rid of: FormatException: Unexpected character (at character 1) error
flutter test test/unit/main_test.dart

# run unit + widget test and convert to junit format
echo "run unit and widget tests"
flutter test --machine | tojunit > TEST-all.xml

cd ..
mv my_app/TEST-all.xml TEST-all.xml

Integration tests

Run tests with the command:

flutter drive --target=test_driver/main.dart > testconsole.log

Parse results and convert them into a Junit .xml file called ‘TEST-all.xml’ or just look at the log file (console.log) after the test run has ended. Move screenshots to the directory in the root called ‘screenshots’:

cd ..
mkdir -p screenshots
mv /tmp/screenshots/test/ screenshots

How to create and start a Flutter test run in Bitbar Cloud

1. Create a zip-file containing app directory (app and tests, in my case ‘my_app’ directory) and ‘run-tests.sh’ file.

2. Select Android as your target OS type and select the ‘Appium Android Server Side’ type as the framework.

3. Upload your test files to the ‘Appium Server Side’ type project in Bitbar Cloud.

Note: Since the actual Flutter app to test is written in the ‘flutter-tests.zip’ file and will be built during the test run, you could upload any dummy .apk file, e.g. ‘bitbar-sample-app.apk’ (see below) to get the test run started on Bitbar Cloud.

4. Start your Flutter test and get the test results

Conclusion

Testing Flutter apps in Bitbar Cloud is available on real Android devices. Integration test results are not showing correctly without parsing and formatting them yourself. Note that examples in this article have nothing to do with the Appium test automation framework.