Why End-to-End Mobile Testing Is A Must
Everywhere you look today, people are on their phones. Whether they’re using Waze to navigate around traffic, checking the weather on their way out the door, ordering food or simply posting to Facebook, people are using mobile devices from sun up to sun down.
The sheer size and potential of the mobile application market is forcing companies across the globe to build and develop mobile software. This attempt to stay competitive and drive new users and customer retention, has given mobile a presence across every industry, from banking to gaming, automotive, and even athletics and hospitality.
The usage of mobile apps has grown exponentially over the past few years and as the demands for mobile development rise, so does the need for mobile testing.
Regardless of the device consumers use, whether they’re an avid Android user or a loyal Apple customer, they expect their mobile applications to function properly. This means every gesture should result in the action they were aiming for. There should be a direct correlation to what they see and what they can touch on the screen. Browsing and submitting an order on Amazon, booking an AirBnB, or depositing a check with their banking institution should all be seamless user experiences. When the gestures and basic functions behind actions like these, such as swipes, clicks, API calls, and data storage don’t work, it can be frustrating or embarrassing to the user.
Many mobile devices and applications also store or access personal data. Apps developed by the banking industry are entrusted with our most confidential financial information. Consumers assume their data and usage is secured and private and when these don’t work, it can be detrimental to both their lives and the company.
This is why mobile testing is critical. The vast array of devices available, high user expectations, and access to sensitive data make the practice vital to the success of any business that decides to develop and market a mobile app.
A good mobile testing strategy will envelop both front and back-end testing to avoid as many issues as possible from causing havoc. API calls need to return specific information and should be secure. The user interface (UI), especially for a smartphone, should be responsive to the device size and respond to gestures such as taps and swipes.
Keep reading this ebook to learn how to build an effective end-to-end testing strategy for mobile development. You will discover:
- The pros and cons of each type of mobile application
- How to overcome the top API and UI mobile testing challenges
- Key features every mobile testing solution should have
- Steps for building an end-to-end mobile testing framework
Types of Mobile Applications
Mobile applications are types of software programs designed to run on smartphones or tablets, such as iPhones, iPads, Android devices, and e-readers. What makes these different from desktop and even web applications are how consumers purchase, download, and interact with them.
Unlike web applications, mobile apps don’t always need to be accessed through browsers or via the internet, and unlike desktop apps, they often have less data storage and bandwidth. Users can access information on their mobile devices instantly while on the go, whereas with desktop and desktop web applications, users are typically rooted in one location. Most notably, mobile devices are touch sensitive.
The shelf life of the average smartphone is also only two or three years. With new versions and products being released regularly, this means users could be on any number of devices.
These differences, along with the myriad of device sizes, operating systems, and versions in the market make mobile testing and development particularly tricky. Which type of mobile app is designed to be the most successful?
There are three different types of mobile applications, each with their own pros and cons.
Native Mobile Apps
Native mobile applications are built with a particular operating system, such as iOS or Android, in mind. They are typically written in Swift (the language of Apple) or Java, for Android devices.
Pros of Native Mobile Apps
- Native mobile applications have to be downloaded from the app store, giving businesses more visibility with end users and making it easier for the app to be discovered.
- Native mobile applications also have the ability to push notifications to users. Whether it’s a gaming or calendar app, businesses can send users reminders and hints, ultimately increasing usage or dwell time – the amount of time consumers spend interacting with an app.
- End users can also access these applications without internet, which makes them easier to access and it won’t increase spend on their phone bill.
- Although devices can be stolen or data can be stored on the device itself, native mobile applications are considered to be the most secure out of all three options as they can leverage frameworks and features that are built into the platform they live on.
Cons of Mobile Native Apps
- Native mobile applications are typically built for usage on a particular operating system. This means the code the app is written in will be dependent on the platform. If you’re looking to have your app be supported on more than one mobile device, this can make maintenance a headache.
Mobile Web Apps
Pros of Mobile Web Apps
- Since end users access mobile web apps via browsers, they don’t have to download the application on their device, meaning it won’t take up storage space on their phone.
- Mobile web apps are often more affordable to develop and maintain as you can leverage cross-platform frameworks like Xamarin.Forms to accelerate your mobile development and testing cycles
Cons of Mobile Web Apps
- While mobile web applications can be found in generic Google search results, they either have to go through an extensive approval process to appear in app stores, or aren’t listed at all.
- Since these apps are heavily dependent on browser access, they can’t be used offline.
Mobile Hybrid Apps
Mobile hybrid applications combine the best features and capabilities of mobile web and native apps. These will load instantly, work offline, and can be installed like a native app.
Pros of Mobile Hybrid Apps
- Compared to native applications, mobile hybrid apps are more affordable to develop and maintain because they are built in a similar fashion to websites.
- This also leads to them having faster load times and there is no need for end users to download updates. Any new features can be pushed live immediately.
- Mobile hybrid applications can be downloaded like a native app or accessed via a browser. This means the software doesn’t have to use a mobile device’s memory or data storage.
- This flexibility also means that mobile hybrid applications can be built on cross-platform frameworks and accessed across a wide variety of mobile environments, browsers, and operating systems.
Cons of Mobile Hybrid Apps
- Mobile hybrid applications tend to run slower and have lower performance especially when compared to native applications since they are not using device storage and depend on caching to increase load times.
- Compared to alternatives, Hybrid apps are considered the least secure. Because hybrid apps depend on a WebView wrapper and leverage caching, they are more vulnerable to data and privacy issues.
Which Mobile App Type Should You Build?
At this point, if you’re wondering which mobile application to develop, or if you’re second guessing the route you’re currently taking, there is no right answer. Progressive web apps are growing in popularity due to their ease of use and development, but every app has its purpose.
Native applications are great for those interested in building mobile games; whereas, mobile web applications are great for enterprise-sized companies that absolutely need their app to be accessible on as many devices as possible.
Regardless of which type you decide to build, you need a solution that can get you across the finish line and provide full coverage across both the front- and back-end of your application.
Finding this solution is the biggest challenge mobile teams face today. Beyond choosing the type of mobile app, it’s essential teams have an end-to-end testing strategy in place for a path forward to test both the front and back-end of their mobile applications.
Barriers to Successful Mobile Testing
Testing the UI and API layers of an application are individually tricky processes, but should both be addressed to build a functional mobile application. To do so successfully, there are a few challenges that need to be overcome.
Challenges with API Mobile Testing
The top API performance issues stem from device fragmentation. There are more than 24,000 unique Android devices in the market today and unlike Apple, Android manufacturers such as Samsung and Motorola, are less consistent in releasing software updates and sunsetting operating systems (OS). Android updates tend to have a much lower adoption rates than Apple’s, with only 1.2% of Android devises using the latest OS in 2017.
This means companies need to ensure their apps function not only across different browsers and device types that have their own screen sizes and resolutions, but also across the different versions of operating systems that have their own user capabilities.
This type of fragmentation often leads to performance issues that are specific to individual mobile platforms, meaning QA teams and developers have to account for each and every unique scenario for complete test coverage.
Security is another key challenge API mobile testers face and is one of the top concerns that companies and users share. Since mobile devices and applications store and access sensitive personal information, it’s vital that security measures are implemented to prevent leaks or hackers from accessing it.
API security testing is all about ensuring you have strong encryptions and strong hosting controls so you’re not exposing your business’ or your users’ data.
Different APIs will require a different degree of security. If you’re building an app for gaming, you probably don’t need to take the same precautions as if you were developing an API for transferring medical or financial data. Regardless, all APIs should be verified and proper security protocols and tests should take place to ensure only the right people can access the data.
Data handling ties directly into your API security measures. There are a few data handling best practices that you can follow to avoid hacks and leaks and build and maintain user trust.
- Collect only the most critical end-user information you need to meet your end users’ needs. If you’re developing a gaming app, you probably only need a name or username, password, and maybe an email address. What you don’t need is height, weight, eye color and a fingerprint. This is an exaggeration, of course, but you can’t lose something you’ve never obtained.
- Request the bare minimum API calls to handle sensitive information. The fewer there are, the less chances there are for a mobile test to miss a bug. On the code level, you want to make sure the UI elements and SQL short codes are secure.
- Ensure detailed data-handling policies are in place. These should describe the data that is being collected and used, who is allowed to access that data, and how it’s being accessed and used.
Challenges with UI Mobile Testing
In a similar fashion to API mobile testing, device fragmentation is a top challenge for UI mobile testing. Device operating systems, hardware specifications, screen sizes, resolutions, and browsers all differ from version to version. This means that mobile native and web applications need to be responsive to the platform they’re being accessed on.
Whether conducted manually or with automation, mobile testing is complex and time-consuming simply due to the myriad of mobile devices in the market. Any good UI test solution can be made easier and provide more coverage with cross-browser and parallel testing.
Device labs are an environment where physical devices such as various smartphones, tablets are held or accessed remotely to those testing a single test script across a variety of configurations. Device labs can alleviate some of the UI testing issues that QA teams face as a result of device fragmentation, but they pose a challenge in and of themselves.
Building and maintaining a device lab that hosts every device you aim to test is expensive and time consuming. Not everyone has the budget or bandwidth to host 24,000 Android devices, much less 100 or even 20. It can be hard to gauge which devices will be used the most in a year or two, especially for Android developers.
While having a device lab gives teams on-demand access to real devices and browsers, there are cloud-hosted laborites that maintain and update real devices at a remote location. This lifts the time and cost burdens associated with building and updating one in-house.
To effectively conduct mobile testing today, it’s essential teams ensure their mobile UI functions and looks as expected across the broadest array of environments as possible.
Easy-to-Use for All Experience Levels
If conducting UI mobile testing, the tool should enable you to visualize and record tests, view test results across configurations, provide checkpoints, as well as the run tests across device loops.
End-to-End Testing Enablement
Look for a tool or vendor that can perform end-to-end testing, especially if you’re responsible for validating both the front and back-end of an application.
For UI testing, good object recognition is vital as it will enable you to build scalable and maintainable automated tests. A tool with robust object recognition capabilities will enable your team to create tests scripts that can more easily identify UI objects and controls across various environments, know when objects are added or removed, and can track object properties that have been changed. This will drive the number of flaky UI tests down and streamline GUI testing as a whole.
Having cross-platform testing capabilities is also critical. Device fragmentation is a global problem for QA teams, and in order to save the time and costs associated with maintaining an in-house device lab to ensure proper coverage, the tool should enable testers to write one piece of code that can be run across multiple environments.
On the API layer, you want your testing tool to be able to access mobile API calls and come with build-in security testing scans.
Access to a Diverse Set of Test Approaches
We all get excited about new releases and new features, but prior to diving into your API or UI testing cycle, you need to outline your test approach. Ask yourself, “Do I need to test on real devices, or do I plan to test on emulators or in the cloud?” The lack of access to any of these will limit what you can do, how long it will take, and how much the process will cost.
Robust Reporting Capabilities
It’s essential you understand how customers use your application and align your tests with real use cases and business scenarios so your testing efforts aren’t wasted. However, your efforts will be for nothing if you can’t report on the efficacy of those tests.
With your API or UI testing tool, you should not only be able to see which tests passed or failed, but you should also be able to pinpoint why. The quicker you can identify a bug, the faster your development team can fix it.
This is where comprehensive reporting will come in handy. Being able to record videos or take screenshots during test runs will enable you to make visual comparisons for cross-platform tests and will help you uncover exactly where they failed.
What about manual tests? Manual testing is still an important practice, especially for UI testing. If the UI of your application is continuously changing and the application is still being upgraded regularly, manual testing might be necessary.In this case, to provide the full picture of your testing efforts, you need both your manual and automated test results to roll into one location.
Continuous Delivery Integrations
Restructuring your existing delivery ecosystem is nearly impossible, so it’s important that the tool you choose is flexible and can adapt to your existing infrastructure. If your developers are becoming more involved in testing, you’ll need a QA tool that integrates with, or can import tests from your developers’ IDE like Visual Studio.
If you’ve already adopted tools like Jenkins or Bamboo, the mobile testing tool you choose should integrate. This will facilitate a faster continuous testing process by allowing commits to be tested faster, feedback to be provided quicker, and for more transparent testing processes across your team.
Mobile Testing in an IoT World
Today, phones do more than make calls, send texts, host games, and connect us through social media. Many mobile devices have built-in GPS receivers that can determine the geographic location of the user and is accessed in mobile applications for maps, navigation, tracking and cartography.
Mobile phones and tablets also come with built-in sensors, such as accelerometers (axis-based motion sensing devices), gravity sensors, light sensors, and magnetometers (instruments used for measuring the earth’s magnetism). For example, an app that can provide compass-bearing information or the temperature will utilize these types of sensors.
If you’re developing applications that use built-in mobile features like these, you need processes in place to validate those systems and ensure that the data is being processed correctly. Incorrectly telling a backpacking tourist they are a mile from where they actually are will cause quite a bit of confusion and frustration.
This is why geolocation testing is critical. To test an application’s behavior in a location that differs from the physical space its taking up, or if you’re using emulators and virtual machines that don’t have a GPS receiver, you can simulate GPS sensor values by specifying what’s known as a mock location, or a predefined location that overrides the actual coordinates. As for sensor data, it may come as no surprise, but different device models have a different sets of sensors, and therefore different capabilities.
The mobile testing tool you choose should be able to conduct cross-platform geolocation testing and needs to be able to ensure sensor data is being processed correctly across those same devices. From your phone, you can now feed your pets remotely, find your friends, and even track how well you brush your teeth.
With our phones and tablets being connected to the vast universe of devices and platforms known as the Internet of Things, mobile development and testing is becoming more and more essential.
Tips for Being Successful with Mobile Testing
Out of all the features and capabilities any good API or UI mobile testing tool should have, there are two you should never make tradeoffs for -- being able to run tests across different mobile environments and adapting tests to specific devices. Follow the two tips below to survive the mobile application landscape.
Set up Device Loops
Set up a device loop to enable faster cross-platform testing. This will enable you to create one test and run it across several connected mobile phones or tablets. The loop will automatically select the mobile device one after another and execute all of the child operations as well. Depending on the tool you choose, you could also leverage multi-device playback capabilities. Record a test on one platform in one environment, and play it back across multiple other devices with the same operating system.
Adapt Mobile Tests to a Specific Device with Name Mapping
Despite your best efforts, not every mobile test created will be able to run on other mobile devices. There are three reasons this can happen:
- Flex layouts: An application may rearrange itself differently based on the layout differences between smartphones and tablets. It may reposition its controls depending on the device orientation, screen size and density.
- Different hierarchy of objects: Each version of an operating system can introduce new objects, variables, and functions. This change causes object hierarchies to change and therefore, increase test maintenance on automated tests.
- Coordinate-Based Actions: Absolute coordinates are not reliable since they depend on the screen resolution of the mobile device. In order to improve your test, you will need to choose a tool that allows you to easily modify it or at least replace absolute coordinates with relative coordinates.
Mobile testing tool features like name mapping will enable you to work around these differences. The goal here is to adjust the mapping criteria in order to uniquely identify the needed objects on any mobile device or search for the needed object.
5 Steps for Building a Mobile Testing Framework
Before diving into your mobile testing strategy and implementing any automated or manual tests, it’s important to build your testing framework. Testing frameworks are essential for any testing process. Built correctly, they can help teams lower maintenance costs, maximize test coverage, improve test efficiency by making code reusable.
Regardless of where you sit in the SDLC, the testing framework you build will ensure you have proper end-to-end mobile testing coverage by incorporating both the API and UI layer.
Make sure you check off each box in the following workflow before testing your mobile application.
At SmartBear, we develop the top software quality tools for teams looking to build extraordinary products. From API design and definition, to back and front-end automated testing, performance monitoring, and code review, we’ve got your mobile testing needs covered.
ReadyAPI: Rest & Soap Testing for APIs
ReadyAPI is a comprehensive API testing platform that enables QA teams to write, run, integrate and automate advanced API functional, performance, and security tests.
Ensure your mobile applications function as intended and call the right information every time with SoapUI Pro, the world’s most widely-used testing tool for SOAP and rest APIs. With SoapUI Pro, you can data-drive input and validation.
Are you developing a mobile app that handles sensitive data? SoapUI Pro will help you find and address any API security concerns you may have by checking that any potential vulnerabilities don’t make it to production. The tool’s mobile security testing capabilities enable you to design and execute custom security tests or leverage several native security scans that can be easily added to your API tests. These include:
- Boundary Scans
- Cross-site scripting
- Fuzzing scan
- Invalid types
- Malformed XML
- Malicous attachments
- SQL injection
- XML Bomb
- XML injection
TestComplete: UI Testing for Mobile Native & Hybrid Apps
TestComplete is an automated UI testing tool for desktop, web and mobile applications. The tool’s mobile testing capabilities enables teams to create and run repeatable GUI tests across native or hybrid iOS or Android apps.
Whether you’re looking to run mobile UI tests on real devices, emulators or virtual machines, you can choose what you need or have access to all three with TestComplete - all without having to jailbreak your phone or tablet.
With integrations to tools such as Jenkins, Git, and Jira, or SoapUI, you can seamlessly weave mobile tests into any continuous delivery environment and conduct end-to-end testing for full mobile coverage.
CrossBrowserTesting: UI Testing for Mobile Web & Hybrid Apps
CrossBrowserTesting is an all-in-one cloud testing platform that enables teams to run UI tests across any web or mobile web application. Using Selenium or Appium for web testing? Not a problem. With the tools optimized cloud-testing infrastructure, you can run your Selenium and Appium tests in parallel across hundreds, if not thousands of browsers and mobile devices concurrently to cut your test execution time down to a fraction of what it is.
The tool offers a wide variety of real and physical devices for manual, visual and automated testing. With instant access to 1500+ real desktop and mobile browsers, you can ditch your device lab for no-hassle and no-maintenance cross-platform GUI testing to expand test coverage.
With integrations to your favorite frameworks, like WebDriver.IO, Protractor, or NightwatchJS, or CI/CD platforms like Jira, GitHub, and Jenkins, you can test continuously for higher quality software and faster deployments.
CrossBrowserTesting provides access to developer tools and extensions like FireBug, as well as top-of-the-line visual testing capabilities. You can debug, fix issues, and share results with the entire team as quickly as possible– ultimately creating faster feedback looks and facilitating better cross-team collaboration.