Strive for Hermetic Testing But Never Compromise Integrity of App

  November 27, 2013

In this blog we’ll take a look at two basic rules of mobile app testing: 1) why it is important to create tests that are fully separated from your application, and 2) how to create a hermetic app testing approach that allows you to isolate your app from external dependencies, speed up the testing, but still providing the real-world context – and the user experience.

Several things change while your app is getting implemented: The user interface and usability, graphics, overall functionality, new language supports, privacy policies, use of external resources, and many other things. It is critical to understand that even if 10% of your code changes or some new features are added to the app, you still need to test 100% of features for regression. More than ever, having a solid testing strategy – and properly built test cases – are vital in the success of your app.

Today, most of the mobile services have a server-client interaction, requiring a login, uploading of data (e.g. pictures, video) and downloading of data (e.g. news, data, images). When developing these kinds of services to be part of your app, developers need to ensure that all their changes are done in code, to either server-side or client-side do not break the functionality of the service. Furthermore, this type of environment can be also set up on a local machine/server where critical connectivity and interaction between user and server can be tested – like it would be done from our app. In fact, testing for some of these types of use cases could be done efficiently with a hermetic environment – however, not jeopardizing the integrity of your app.

Testing in both

Separate your tests from the app itself

Ideally, tests should have full access inside an application and test all aspects of it: memory contents, data tables, file contents, and internal states to determine if the app entity is behaving as expected. In fact, the executed tests will reveal what needs to be changed in your app – not vice versa so that you need to tweak the app for testing. With the Android Instrumentation framework, for instance, this is straightforward and easy.

Android provides the testing API that provides so-called “hooks” into the Android component and application life cycle. These hooks are called the instrumentation and allow your tests to control the application, its life cycle, user interactions, and also other events. Typically, the app can only respond to these events but with instrumentation, users can control these events through their tests. An instrumentation-based test class allows you to send events to the application under test.

The following picture – as seen and presented by Google – concludes the Android testing scheme and illustrates how different pieces in the testing framework are related to each other:

android testing scheme presented by google

The Android instrumentation API allows users to run the test project and the normal Android project in the same process so that the test project can call methods of the Android project directly. Based on this sort of philosophy, the test cases/scripts are separated from the application and doesn’t cause a need to do tweaks on your app for testing.

The InstrumentationTestRunner is the primary Android test runner class. It extends the JUnit test runner framework, is also instrumented, and it can run any of the test case classes provided by Android. Basically, the InstrumentationTestRunner starts and loads the test methods, and communicates with the Android system level through the instrumentation API. As the test method controls the life cycle of the components in the application, it will also start the application for testing.

Try to create a hermetic app testing environment – if possible

By minimizing the external dependencies from your app testing the setup can help you to focus on testing the most essential things in your app. For example, if you need to test how your app reacts with certain front or back-end entity, there are several ways to simulate this sort of behavior. You actually do not need to have a physical server or even network connection to run your tests but you can build this type of localized service as part of your tests.

A hermetic environment for app testing is typically running without any external dependency or external service (e.g. phone network) calls. In order to run tests in a hermetic environment, a white-box testing framework (e.g. Espresso) can be ideal for your needs. This new framework – still in early-phase – will synchronize your tests and actions with events. However, it is worth mentioning that this new framework is very UI-focused but still very useful for app testing.

For instance, tests involved with network access of any type are slower than the ones that access local resources. You can enable localhost (your computer) to run the same services that would be provided for users through a network connection. External service can be started upon a single machine that has no network connection. As testing (and implementing) safety-critical applications on in-house environments is a must requirement, hermetic test environments offer a valid approach to enable efficient testing on these sorts of apps.

Bitbar solution for hermetic app testing

Example: In-house testing platform

So-called “private-cloud” can help app developers to create an in-house testing platform – and together with agile/continuous integration automate the mobile app testing effort. It provides a very efficient way to test every build – even smaller changes in your app – together with your development. One of the great benefits of this type of development and testing approach is that it integrates also nicely to back-end services that can be built locally.

This approach supports running any  JUnit tests, all possible test automation frameworks as well as Compatibility Test Suite (CTS) for Android, and easy integration of any third-party frameworks/scripts to your daily build/test environment.