Mastering the Art of Mobile Testing

Everywhere you look today, people are on their phones. Whether they’re using Waze to navigate around traffic, checking the weather on their way out the door, ordering food, or simply posting to Facebook, people are using mobile devices from sun up to sun down.

The sheer size and potential of the mobile application market is forcing companies across the globe to build and develop mobile-first software. This attempt to stay competitive and drive both new users and customer retention, has given mobile a presence across every industry, from banking to gaming, automotive, and even athletics and hospitality.  

The usage of mobile apps has grown exponentially over the past few years and as the demands for mobile development rise, so does the need for mobile testing.

Regardless of the device consumers use, whether they’re an avid Android user or a loyal Apple customer, they expect their mobile applications to function properly. This means every gesture should result in the action they were aiming for. There should be a direct correlation to what they see and what they can touch on the screen. Browsing and submitting an order on Amazon, booking an AirBnB, or depositing a check with their banking institution should all be seamless user experiences. When the gestures and basic functions behind actions like these, such as swipes, clicks, API calls, and data storage don’t work, it can be frustrating or embarrassing to the user.

Many mobile devices and applications also store or access personal data. Apps developed by the banking industry are entrusted with our most confidential financial information. Consumers assume their data and usage is secured and private and when these don’t work, it can be detrimental to both their lives and the company.

This is why mobile testing is critical. The vast array of devices available, high user expectations, and access to sensitive data make the practice vital to the success of any business that decides to develop and market a mobile app.

A good mobile testing strategy will envelop both front and back-end testing to avoid as many issues as possible from causing havoc. API calls need to return specific information and should be secure. The user interface (UI), especially for a smartphone, should be responsive to the device size and respond to gestures such as taps and swipes.

Keep reading this ebook to learn how to build an effective end-to-end testing strategy for mobile development. You will discover:

• The pros and cons of each type of mobile application
• How to overcome the top API and UI mobile testing challenges
• Key features every mobile testing solution should have
• Steps for building an end-to-end mobile testing framework

Before diving into your mobile testing strategy and implementing any automated or manual tests, it’s important to build your testing framework. Testing frameworks are essential for any testing process. Built correctly, they can help teams lower maintenance costs, maximize test coverage, improve test efficiency by making code reusable. 

Regardless of where you sit in the SDLC, the testing framework you build will ensure you have proper end-to-end mobile testing coverage by incorporating both the API and UI layer.

Make sure you check off each box in the following workflow before testing your mobile application.

Testing the UI and API layers of an application are individually tricky processes, but should both be addressed to build a functional mobile application. To do so successfully, there are a few challenges that need to be overcome.

Device Fragmentation:

In a similar fashion to API mobile testing, device fragmentation is a top challenge for UI mobile testing. Device operating systems, hardware specifications, screen sizes, resolutions, and browsers all differ from version to version. This means that mobile native and web applications need to be responsive to the platform they’re being accessed on.

Whether conducted manually or with automation, mobile testing is complex and time-consuming simply due to the myriad of mobile devices in the market. Any good UI test solution can be made easier and provide more coverage with cross-browser and parallel testing.

Usability & Gestures

One of the challenges of mobile testing is also one of the reasons for mobiles explosive growth - touch screens. Touch screens bring a whole new level of complexity to testing, requiring special automation tools, like Appium or TestComplete Mobile, to mimic user actions at the GUI level.

Networks & Connectivity

There are several different types of connection protocols (Edge, 3G, 4G, LTE, etc) that can complicate testing. Also, because native mobile applications can be accessed without the internet, there will be scenarios where connectivity is lost but features must seamlessly work.

Battery Life & Performance

Normally, web and desktop applications are accessed by devices plugged into a power source or with large external batteries. Mobile devices have limited battery storage, meaning that applications must be highly optimized to save resources.

Device Labs:

Device labs are an environment where physical devices such as various smartphones, tablets are held or accessed remotely to those testing a single test script across a variety of configurations. Device labs can alleviate some of the UI testing issues that QA teams face as a result of device fragmentation, but they pose a challenge in and of themselves.

Building and maintaining a device lab that hosts every device you aim to test is expensive and time consuming. Not everyone has the budget or bandwidth to host 24,000 Android devices, much less 100 or even 20. It can be hard to gauge which devices will be used the most in a year or two, especially for Android developers.

While having a device lab gives teams on-demand access to real devices and browsers, there are cloud-hosted laborites that maintain and update real devices at a remote location. This lifts the time and cost burdens associated with building and updating one in-house.

To effectively conduct mobile testing today, it’s essential teams ensure their mobile UI functions and looks as expected across the broadest array of environments as possible.

Mobile applications are types of software programs designed to run on smartphones or tablets, such as iPhones, iPads, Android devices, and e-readers.  What makes these different from desktop and even web applications are how consumers purchase, download, and interact with them.

Unlike web applications, mobile apps don’t always need to be accessed through browsers or via the internet, and unlike desktop apps, they often have less data storage and bandwidth. Users can access information on their mobile devices instantly while on the go, whereas with desktop and desktop web applications, users are typically rooted in one location. Most notably, mobile devices are touch sensitive.

The shelf life of the average smartphone is also only two or three years. With new versions and products being released regularly, this means users could be on any number of devices. These differences, along with the myriad of device sizes, operating systems, and versions in the market make mobile testing and development particularly tricky. 

To add to this complexity is the fact that there are three different types of mobile applications - each with their own pros and cons. Let's explore how each one positions itself for success in consumer's hands.

CrossBrowserTesting is an all-in-one cloud testing platform that enables teams to run UI tests across any web or mobile web application. Using Selenium or Appium for web testing? Not a problem. With the tools optimized cloud-testing infrastructure, you can run your Selenium and Appium tests in parallel across hundreds, if not thousands of browsers and mobile devices concurrently to cut your test execution time down to a fraction of what it is.

The tool offers a wide variety of real and physical devices for manual, visual and automated testing. With instant access to 1500+ real desktop and mobile browsers, you can ditch your device lab for no-hassle and no-maintenance cross-platform GUI testing to expand test coverage.

With integrations to your favorite frameworks, like WebDriver.IO, Protractor, or NightwatchJS, or CI/CD platforms like Jira, GitHub, and Jenkins, you can test continuously for higher quality software and faster deployments.

CrossBrowserTesting provides access to developer tools and extensions like FireBug, as well as top-of-the-line visual testing capabilities. You can debug, fix issues, and share results with the entire team as quickly as possible– ultimately creating faster feedback looks and facilitating better cross-team collaboration.  

Native mobile applications are built with a particular operating system, such as iOS or Android, in mind. They are typically written in Swift (the language of Apple) or Java or Kotlin, the preferred languages for Android devices.

Pros of Native Mobile Apps

• Native mobile applications have to be downloaded from the app store, giving businesses more visibility with end users and making it easier for the app to be discovered. 
• Native mobile applications also have the ability to push notifications to users. Whether it’s a gaming or calendar app, businesses can send users reminders and hints, in effort to increase usage or dwell time – the amount of time consumers spend interacting with an app.
• End users can also access these applications without internet, which makes them easier to access.
• Although devices can be stolen or data can be stored on the device itself, native mobile applications are considered to be the most secure out of all three options as they can leverage frameworks and features that are built into the platform they live on.

Cons of Mobile Native Apps

1. Native mobile applications are typically built for usage on a particular operating system. This means the code the app is written in will be dependent on the platform. These languages are not as popular as their web focused counterparts.
2. If you’re looking to have your app be supported on more than one mobile device, get ready to expend resources. This can make maintenance a headache as you'll often need to maintain two different code bases. 
3. Updates made to native applications often have to be manually accepted by App store moderators, slowing down updates and deployments.

Progessive Web Apps (Mobile hybrid) combine the best features and capabilities of mobile web and native apps. These will load instantly, work offline, and can be installed like a native app.

Pros of Progressive Web Apps

• The majority of mobile hybrid applications can be written in a standard language like JavaScript, HTML, or CSS and are then run through WebDriver. As with mobile web apps, this means there is typically only one codebase that needs to be maintained. 
• Compared to native applications, mobile hybrid apps are more affordable to develop and maintain because they are built in a similar fashion to websites.
• This also leads to them having faster load times and there is no need for end users to download updates. Any new features can be pushed live immediately.
• This flexibility also means that mobile hybrid applications can be built on cross-platform frameworks and accessed across a wide variety of mobile environments, browsers, and operating systems.

Cons of Mobile Hybrid Apps

1. Mobile hybrid applications tend to run slower and have lower performance especially when compared to native applications since they are not using device storage and depend on caching to increase load times.
2. Compared to alternatives, Hybrid apps are considered the least secure. Because hybrid apps depend on a WebView wrapper and leverage caching, they are more vulnerable to data and privacy issues.

At this point, if you’re wondering which mobile application to develop, or if you’re second guessing the route you’re currently taking, there is no right answer. Progressive web apps are growing in popularity due to their ease of use and development, but every app has its purpose.

Native applications are great for those interested in building mobile games; whereas, mobile web applications are great for enterprise-sized companies that absolutely need their app to be accessible on as many devices as possible and may not need the advanced functionality of native applications.

Regardless of which type you decide to build, you need a solution that can get you across the finish line and provide full coverage across both the front- and back-end of your application.

Finding this solution is the biggest challenge mobile teams face today. Beyond choosing the type of mobile app, it’s essential teams have an end-to-end testing strategy in place for a path forward to test both the front and back-end of their mobile applications.

Mobile web applications are simply websites that have been optimized for mobile browsers and often contain dynamic content or "features" that can perform functions similar to mobile applications. These types of application don't need to be downloaded from an app store and can be coded in standard web languages like JavaScript and HTML5.

Pros of Mobile Web Apps

1. Since end users access mobile web apps via browsers, they don’t have to download the application on a device, meaning it won’t take up storage space on their phone.
2. Mobile web apps can be written in JavaScript, Python, PHP, Ruby, and more - meaning that at the end of the day, there is only one codebase that needs to be maintained. These languages are popular and widely supported.
3. Any updates that need to be made can be pushed live and appear in production instantly - no screening process from a central moderator.

Cons of Mobile Web Apps

1. Since these apps are heavily dependent on browser access, they can’t be used offline.
2. While mobile web applications are progressing, they still lack features of native applications, like access to gyrometer, altimeter, and other hardware functionality.
3. While mobile web applications can be found in generic Google search results, they don't get visibility inside of native app stores.

Today, phones do more than make calls, send texts, host games, and connect us through social media.  Many mobile devices have built-in GPS receivers that can determine the geographic location of the user and is accessed in mobile applications for maps, navigation, tracking and cartography.

Mobile phones and tablets also come with built-in sensors, such as accelerometers (axis-based motion sensing devices), gravity sensors, light sensors, and magnetometers (instruments used for measuring the earth’s magnetism). For example, an app that can provide compass-bearing information or the temperature will utilize these types of sensors.

If you’re developing applications that use built-in mobile features like these, you need processes in place to validate those systems and ensure that the data is being processed correctly. Incorrectly telling a backpacking tourist they are a mile from where they actually are will cause quite a bit of confusion and frustration.

This is why geolocation testing is critical. To test an application’s behavior in a location that differs from the physical space its taking up, or if you’re using emulators and virtual machines that don’t have a GPS receiver, you can simulate GPS sensor values by specifying what’s known as a mock location, or a predefined location that overrides the actual coordinates. As for sensor data, it may come as no surprise, but different device models have a different sets of sensors, and therefore different capabilities.

The mobile testing tool you choose should be able to conduct cross-platform geolocation testing and needs to be able to ensure sensor data is being processed correctly across those same devices. From your phone, you can now feed your pets remotely, find your friends, and even track how well you brush your teeth.  

With our phones and tablets being connected to the vast universe of devices and platforms known as the Internet of Things, mobile development and testing is becoming more and more essential.

At SmartBear, we develop the top software quality tools for teams looking to build extraordinary products. From API design and definition, to back and front-end automated testing, performance monitoring, and code review, we’ve got your mobile testing needs covered.

ReadyAPI is a comprehensive API testing platform that enables QA teams to write, run, integrate and automate advanced API functional, performance, and security tests.

Ensure your mobile applications function as intended and call the right information every time with SoapUI Pro, the world’s most widely-used testing tool for SOAP and rest APIs. With SoapUI Pro, you can data-drive input and validation.

Are you developing a mobile app that handles sensitive data? SoapUI Pro will help you find and address any API security concerns you may have by checking that any potential vulnerabilities don’t make it to production. The tool’s mobile security testing capabilities enable you to design and execute custom security tests or leverage several native security scans that can be easily added to your API tests. These include:

• Boundary Scans
• Cross-site scripting
• Fuzzing scan
• Invalid types
• Malformed XML
• Malicous attachments
• SQL injection
• XML Bomb
• XML injection

TestComplete is an automated UI testing tool for desktop, web and mobile applications. The tool’s mobile testing capabilities enables teams to create and run repeatable GUI tests across native or hybrid iOS or Android apps.

Teams of all skill levels, from automation beginners to experts, can create complex scripts in TestComplete by scripting in popular languages such as JavaScript, Python, and VBScript, or by leveraging features such as robust Record and replay capabilities. You can even increase mobile device coverage by running tests for multi-touch gestures like swipes, drags, and accessing mobile device and GPS sensors.

Whether you’re looking to run mobile UI tests on real devices, emulators or virtual machines, you can choose what you need or have access to all three with TestComplete - all without having to jailbreak your phone or tablet.

With integrations to tools such as Jenkins, Git, and Jira, or SoapUI, you can seamlessly weave mobile tests into any continuous delivery environment and conduct end-to-end testing for full mobile coverage.

Easy-to-Use for All Experience Levels

First and foremost, a good mobile testing tool should be accessible to testers with a variety of skill sets. Anyone on your team should be able to create, run, and maintain API or UI tests, whether they just getting started or are a seasoned automation engineer or developer. This means the tool should support a variety of scripting and programming languages for expert coders, like JavaScript, Python, or Swift, while also providing a scriptless-means of test creation that will still enable beginners to create complex tests.

Cloud Based Device Lab

Building and maintaining a device lab that hosts every device you aim to test is expensive and time consuming. Not everyone has the budget or bandwidth to host 24,000 Android devices, much less 100 or even 20. It can be hard to gauge which devices will be used the most in a year or two, especially for Android developers. There are cloud-hosted labs that maintain and update real devices, so you can focus on development. In order to save the time and costs associated with maintaining an in-house device lab to ensure proper coverage, the tool should enable testers to write one piece of code that can be run across multiple environments.

Robust Object and Gesture Recognition

For UI testing, good object recognition is vital as it will enable you to build scalable and maintainable automated tests. A tool with robust object recognition capabilities will enable your team to create tests scripts that can more easily identify UI objects and controls across various environments, know when objects are added or removed, and can track object properties that have been changed. This will drive the number of flaky UI tests down and streamline GUI testing as a whole.

Robust Reporting Capabilities

It’s essential you understand how customers use your application and align your tests with real use cases and business scenarios so your testing efforts aren’t wasted. However, your efforts will be for nothing if you can’t report on the efficacy of those tests. 

With your API or UI testing tool, you should not only be able to see which tests passed or failed, but you should also be able to pinpoint why. The quicker you can identify a bug, the faster your development team can fix it.

This is where comprehensive reporting will come in handy. Being able to record videos or take screenshots during test runs will enable you to make visual comparisons for cross-platform tests and will help you uncover exactly where they failed.

What about manual tests? Manual testing is still an important practice, especially for UI testing. If the UI of your application is continuously changing and the application is still being upgraded regularly, manual testing might be necessary.In this case, to provide the full picture of your testing efforts, you need both your manual and automated test results to roll into one location.

Continuous Delivery Integrations

Restructuring your existing delivery ecosystem is nearly impossible, so it’s important that the tool you choose is flexible and can adapt to your existing infrastructure. If your developers are becoming more involved in testing, you’ll need a QA tool that integrates with, or can import tests from your developers’ IDE like Visual Studio.

If you’ve already adopted tools like Jenkins or Bamboo, the mobile testing tool you choose should integrate. This will facilitate a faster continuous testing process by allowing commits to be tested faster, feedback to be provided quicker, and for more transparent testing processes across your team.

Out of all the features and capabilities any good API or UI mobile testing tool should have, there are two you should never make tradeoffs for -- being able to run tests across different mobile environments and adapting tests to specific devices. Follow the two tips below to survive the mobile application landscape.

Set up Device Loops

Set up a device loop to enable faster cross-platform testing. This will enable you to create one test and run it across several connected mobile phones or tablets.  The loop will automatically select the mobile device one after another and execute all of the child operations as well.  Depending on the tool you choose, you could also leverage multi-device playback capabilities. Record a test on one platform in one environment, and play it back across multiple other devices with the same operating system.

Adapt Mobile Tests to a Specific Device with Name Mapping

Despite your best efforts, not every mobile test created will be able to run on other mobile devices. There are three reasons this can happen:

1. Flex layouts: An application may rearrange itself differently based on the layout differences between smartphones and tablets. It may reposition its controls depending on the device orientation, screen size and density.
2. Different hierarchy of objects: Each version of an operating system can introduce new objects, variables, and functions. This change causes object hierarchies to change and therefore, increase test maintenance on automated tests.
3. Coordinate-Based Actions: Absolute coordinates are not reliable since they depend on the screen resolution of the mobile device. In order to improve your test, you will need to choose a tool that allows you to easily modify it or at least replace absolute coordinates with relative coordinates.

Mobile testing tool features like name mapping will enable you to work around these differences. The goal here is to adjust the mapping criteria in order to uniquely identify the needed objects on any mobile device or search for the needed object.