What You Should Know About App Testing During Development

Bugs are evil for your mobile applications, games and websites. They surely are the ultimate reason why your app doesn’t get 100 million downloads from app markets. That’s why you need to do app testing. To get the most out of your testing efforts, getting involved with app testing as early as possible in the development phase is the best way to ensure good quality, on time and with less overall effort and costs.

It’s a well-known fact that automated testing methods can be used for both validating requirements and reducing the costs of testing through automated test case generation – when done right and preferably in the earliest phase of app development. However, the full automation also has a cost that many companies haven’t been ready to pay for. Knowing how much fixing a bug later during the development, everything related to testing should be considered at the beginning of your project.

A common understanding of testing has been that any app can be tested once 1) it exists and 2) is stable enough for testing. But the truth is that testing elements in apps and hardware as early as in development phase can bring you enormous benefits.

Software and Hardware

To get started developers should think about not only how to test early but also understand why to do automated testing and how to build a standard testing procedure at the beginning. When it’s properly configured, you can save lots of hours in actual development and bug fixing. The complexity of today’s apps is constantly increasing and customers’ need is demanding. In addition, running tests on several devices or emulators doesn’t guarantee the same results on similar devices.

Various hardware today causes ‘fragmentation’ and difficulty to get the maximal performance out of your app. For instance, the major differences in any chipsets performance capability – including both CPU and GPU – can make your app perform differently on devices based on those chipsets.

Resolution. The resolution of the display is not the only meaningful form factor – the quality, density or color brightness of the display component. All together can have a significant impact on how it shows up to your uses. Based on that, the only way to understand the truth is run automated testing on a wide range of real devices to get the whole picture.

Memory. The memory form factor is significant and has an impact on apps. Due to insufficient memory, low-end devices lack the ability to run large, memory-consuming apps. In many cases, your app users won’t get notified about anything else but “low memory”. At this point, they probably will blame this matter on your app.

Sensors and other components. Sensors vary a lot based on different devices. Most of the devices nowadays incorporate the standard sensors that measure motion, orientations, and various environmental conditions. Accelerometers, gravity sensors, gyroscopes, and rotational vector sensors, as well as other orientation sensors and magnetometers, are typical sensors in today’s devices. For example, games utilize sensor data intensively and in many games, it is just not enough that a sensor exists and is calibrated but it should be responsive in-game context and work to produce the best user experience.

Software. When it comes to OS versions, it’s a totally different story. OS versions are corresponding to different API levels, which contain significant differences with each API level. The other ones – maybe not as visible source as a headache – are the OEM customizations on vanilla Android. Nearly all of Android OEMs have built their own UI layers, skins to empower their brand and differentiate their devices. Basically there is nothing wrong with that for consumers but for developers again it can cause stress as your applications work differently on all of these brands. The only solution is to use a reliable cloud-based testing solution to test your app on a large scale of devices.

Let’s take a look at one example: The different OEM, different OS version, different hardware but identical software:

The very same piece of code executed identically on all three devices.

Testing Apps While Developing

If you start testing a few days before your app hit the market, you’re too late. A wealth of aspects, data and functionality should be tested from Day One. Here are some things to consider how to include testing as a part of your mobile app development:

1. Automate Generic Things as Much as You Can

When building mobile apps, a well-planned testing strategy will greatly benefit you. Great user experience and design are paramount, and solid development and testing are fundamental. Many things in mobile app testing can be automated and those tests can increase the depth and scope of testing and help significantly to improve your app quality. Ideally, test cases should have full access inside an application and test all aspects of it: memory contents, data tables, file contents, and internal program states to determine if the product is behaving as expected.

2. App Changes During Development and Testing

Once you start changing a piece of code, many other things are changing as well, user interface, graphics, functionality, to name a few. Even if you change 10% of app source code, you still need to test your app features in a holistic way. In this case, manual testing can never scale up your, and creating all test cases for new features is your best option. Hereby all test cases can be utilized to do automated testing on your new features. It is essential to building your tests so that those are maintainable during the development phases of your app.

3. Select a Testing Tech and Provider You Can Grow With

If you have an app already at market and are looking to create another similar type of app, make sure you select a technology and vendor provider that you can use for your needs. For example, building your test cases with a certain method or framework means that those test cases could be reused with your new application – at least to some extent. For this it is important to select a technology and vendor that is going to be able to handle your needs as your product scales, testing needs to cover a new geographical area, and even support new platforms (e.g. Android -> iOS).

4. Do 24/7 Testing with Test Automation

Test automation will help you to reduce the time of testing new features and whole app by allowing test execution to happen 24/7.

5. Scale Up with Cloud-Based Testing Platform

A cloud-based testing platform like Bitbar Testing allows you to perform automated testing on thousands of real Android and iOS devices and remotely control our devices to do manual testing. On Bitar Testing, you will not only get access to popular European and North American devices, but also mainstream Asian devices are at your disposal. One test run on our platform can give a thorough overview of how your app performs on various devices from a global perspective.

Testing Types That Can Be Incorporated in Development Phase

Use cases are believed to be one of the best basis to build real-world test cases. For some reason in the past, to automate the test generation process, there has been a large gap in bridging the high-level use cases and concrete tests that hammer your software. Here are some interesting – and yet more important – testing angles to keep in mind when you are planning the testing methods (and tools) for your mobile application, game or service:

1. Functional Automated Testing

The functional testing typically involves several steps to describe what the system does: From the identification of functions that software is expected to perform, a creation of input data and output based on the specification, the actual test case execution and comparison of actual and expected outputs. In order to automate functional testing, it is highly recommended to adopt professional tools and run test cases again real Android and iOS devices.

2. Performance and Stress tests

In order to determine how well your application responds to various levels of usage, users should consider creating tests which last for hours to observe how well the application, as well as device/server combinations, behave during long usage. To determine if your application will be able to run successfully for a sustained duration under heavy load, stress tests can be used. The performance tests will measure how responsive your application can be on a real device.

3. Global Access to Real Devices

Nowadays, the beauty of automated testing for mobile apps is that you don’t even need to own or possess the physical device you are testing your application on. For example, some pre-market devices are located in Hong Kong and CDMA mobile devices are in the US. But you can still have access to these devices, conduct your instrumentation tests and receive results in minutes.

4. Regression Testing

Most of the mobile services today have a server-client interaction, requiring a login, uploading of data (e.g. pictures) and downloading of data (e.g. news, data, images). When developing these kinds of services, users need to ensure that all their changes done in code, to either server side or client side do not break the functionality of the service. Testing for these types of use cases may be practically impossible to be done manually and automated testing is really the only option.

5. Localization Testing

Testing becomes highly important when your application is targeted for the global markets. The word ‘Global’ means that your app or game needs to be ‘local’ for everyone. When your application needs to be translated and tested with devices in multiple languages, these types of tests can be easily performed automatically to change the language of devices and even do it again with the same test runs. The problems layouts can be easily picked from screenshots.

Conclusion

Testing is a crucial activity to ensure success in a highly competitive landscape of mobile apps and games. But poorly planned testing can be 20-50% of the total development effort, and in that case, it’d also be the biggest single cost in your budget. In order to ensure that testing can be done extremely efficiently, covering the essence of today’s mobile ecosystems and different device models, relying on a trustful and robust cloud-based testing platform, like Bitbar Testing is the best option.