Test Early, Test Often, Testing as Part of App Development

The quality of mobile applications is crucial – and should be always monitored. Frankly, you should be extremely interested in the quality of your soon-to-be-published or updated application at Google Play or App Store. If your app delivers a bad user experience those disappointed users won’t be kind and get you out of the hook – no matter how much they have used/liked/loved your app in the past. The earlier you test, the earlier you find faults, and the earlier you fix those problems and win the trust of users. To achieve this, incorporating testing as part of app development is utmost important.

Regardless from which point of view you or your app’s users are taking, bugs are evil and at the very core of why your mobile application or game doesn’t get those 100 million downloads from markets. To get the most out of your testing efforts, getting involved with app testing as early as possible in the development phase is the best way to ensure good quality, on time and with less overall effort and costs.

It’s a well-known fact that automated testing methods can be used for both validating requirements and reducing the costs of testing through automated test case generation – when done right and preferably in the earliest phase of the development. However, the full automatization also has a cost that many companies haven’t been ready to pay for. Knowing how much fixing a bug later during the development costs, everything related to testing should be considered at the beginning of your project.

A common understanding of testing has been that any app can be tested once 1) it exists and 2) is stable enough for testing. In this blog, we’re taking a look at some elements in apps and hardware that can be tested in any phase of the development, how to include testing in your development and what type of testing can be done in each phase.

Getting Started – Think About Both: Software and Hardware

To get started developers should think about not only how to test early but also how to build a standard testing procedure at the beginning. When done properly it can cumulate as a massive saving of hours in the actual development and bug fixing. The complexity of today’s apps is constantly increasing and the case for testing becomes more relevant and imminent. In addition, the device landscape is very versatile – running your app on a certain version of the operating system doesn’t guarantee that your app runs fine on all similar devices with the same OS version.

It is also important to notice that not all problems faced by the developer are related to software. Hardware problems cause similar ‘fragmentation’ and difficulty to get the maximal performance out of your app. For example, the major differences in any chipset’s performance capability – including both CPU and GPU – can make your app perform differently on devices based on those chipsets.

The resolution of the display is not the only meaningful form factor – the quality, density or color brightness of display component itself can make your app look bad. There is no other way to observe this but do testing on real devices to understand how badly your app shows up to users.

The same thing applies to memory. Many low-end devices don’t have that much of memory and therefore the ability to run large, memory-consuming apps on those devices is difficult. The memory form factor is significant and has an impact on apps. In so many cases user won’t be notified anything else but “low memory” and down goes your app. Who will users blame in a case like this – your app or the device? You got it right.

Sensors and other components vary a lot based on a different device. Today most of the devices incorporate the standard sensors that measure motion, orientations, and various environmental conditions. Accelerometers, gravity sensors, gyroscopes, and rotational vector sensors, as well as other orientation sensors and magnetometers, are typical sensors in today’s devices. For example, games utilize sensor data intensively, and in many games, it is just not enough that a sensor exists and is calibrated but it should be responsive in-game context and work to produce the best user experience.

And then there is a software side of things. The OS version is only one part of the story. There are significant differences between different API levels and it is a known issue for developers. The other ones – maybe not a visible source of a headache – are the OEM customizations on vanilla Android. Nearly all of Android OEMs have built their own UI layers, skins to empower their brand and differentiate their devices. Basically there is nothing wrong with that for consumers but for developers again it can cause stress as their stuff things to work differently on all of these brands.

Let’s take a look at one example: The same brand, different OS version, different hardware but identical software:

The very same piece of code executed identically on all three devices. There is no way to find it out without testing your software on real devices – and it can be easily done on Testdroid Cloud.

Include Testing as Part of Your App Development

If you start thinking about testing a few days before your app will hit the market, you’re too late. There is a wealth of aspects, data and functionality that should be tested from day one.

1. Plan Carefully – Automate Generic Things as Much as You Can

When building a mobile app, well thought strategy is critical, great user experience and design are paramount, and solid development and testing are fundamental. Many things in mobile app testing can be automated and those tests can increase the depth and scope of testing and help significantly to improve your app quality. Ideally, test cases should have full access inside an application and test all aspects of it: memory contents, data tables, file contents, and internal program states to determine if the product is behaving as expected.

2. During the Development Your App Changes – Same Applies for Testing

Many things change while your app is getting created. The user interface, graphics, functionality, language support, privacy policies, use of external resources, and many other features. It is very important to understand that even if 10% of your code changes or is added to the app, you still need to test 100% of features. Manual testing couldn’t keep up anyhow so the best option is to build all your test cases for new features. When a new feature is added, all features are automatically tested.  It is essential to building your tests so that those are maintainable over the development phases of your app.

3. Select a Testing Tech and Provider You Can Grow With

If you have an app already at the market and are looking to create another similar type of app, make sure you select technology and vendor providing that you can use for your needs. For example, building your test cases with a certain method or framework means that those test cases could be reused with your new application – at least to some extent. For this it is important to select a technology and vendor that is going to be able to handle your needs as your product scales, testing needs to cover a new geographical area, and even support new platforms (e.g. Android -> iOS).

4. Test Automation is for You 24/7

Test automation will help you to reduce the time of testing new features and a whole app by allowing test execution to happen 24/7. If you use a cloud-based mobile testing platform, make sure you have access to 24-hour support.

5. Utilize Cloud-Based Testing Platform – Get Real Global Reach

With a cloud-based mobile app testing service, you can get instant access to over 250 real Android devices and over 15 different iOS variants. Especially in case of Android, devices are hand-picked from the global marketplace so that they provide you a significant coverage in volumes. Running automated tests on those devices is easy, fast and provide you all information from the real device execution – pre-processed, summarized and with full details.

What Type of Testing Can Be Included As Part of App Development Phase?

Use cases are believed to be one of the best basis to build real-world test cases. For some reason in the past, to automate the test generation process, there has been a large gap in bridging the high-level use cases and concrete tests that hammer your software. Here are some interesting – and yet more important – testing angles to keep in mind when you are planning the testing methods (and tools) for your mobile application, game or service:

1. Enhance your Quality Assurance with Functional Automated Testing.

The functional testing typically involves several steps to describe what the system does: From the identification of functions that software is expected to perform, the creation of input data and output based on the specification, the actual test case execution and comparison of actual and expected outputs. In order to automate functional testing, it is highly recommended to adopt professional tools and run test cases again real Android and iOS devices.

2. Improve Responsiveness of Application with Performance and Stress tests.

In order to determine how well your application responds to various levels of usage, users should consider creating tests which last for hours to observe how well the application, as well as device/server combinations, behave during long usage. To determine if your application will be able to run successfully for a sustained duration under heavy load, stress tests can be used. The performance tests will measure how responsive your application can be on a real device.

3. Enabling Global Access to Real Devices, either Stock or Pre-Market Devices

Nowadays, the beauty of automated testing for mobile apps is that you don’t even need to own or possess the physical device you are testing your application on. For example, some pre-market devices are located in Hong Kong and CDMA mobile devices are in the US. But you can still have access to these devices, conduct your instrumentation tests and receive results in minutes.

4. Uncover New Software Bugs with Regression Testing

Most of the mobile services today have a server-client interaction, requiring a login, uploading of data (e.g. pictures) and downloading of data (e.g. news, data, images). When developing these kinds of services, users need to ensure that all their changes are done in code, to either server-side or client-side do not break the functionality of the service. Testing for these types of use cases may be practically impossible to be done manually and automated testing is really the only option.

5. Ensure Your “Global” App is Understood Locally

Testing becomes highly important when your application is targeted for the global markets. The word ‘Global’ means that your app or game needs to be ‘local’ for everyone. When your application needs to be translated and tested with devices in multiple languages, these types of tests can be easily performed automatically to change the language of devices and even do it again with the same test runs. The problems layouts can be easily picked from screenshots.

Conclusion

Testing is a crucial activity to ensure success in a highly competitive landscape of mobile apps and games. But “poorly” planned testing can be 20-50% of the total development effort, and in that case, it’d also be the biggest single cost in your budget. In order to ensure that testing can be done extremely efficiently, covering the essence of today’s mobile ecosystems and different device models, Testdroid is the only, smart option available.