How to Create a Winning Mobile Testing Strategy
With the conception of responsive web design, it’s become easier to build and enjoy the mobile web on any screen or device. As more and more users carry higher expectations from their mobile experiences on an increasing number of smartphones, a mobile testing strategy is no longer a competitive advantage, but an essential part of your overall QA process.
That’s why we’ve prepared this guide that includes the four steps to lead you to a successful mobile test strategy. By better understanding the necessary parts of mobile testing, you can be more prepared to create an exceptional mobile experience for every user.
1. Gather Requirements
The first step to a mobile testing strategy, or really any testing strategy, is to gather requirements. The difference here is that the product manager will likely have separate requirements when it comes to mobile users, so it’s important to understand what those are. A big reason for this is because your mobile users will likely have different testing personas from your normal desktop users and will probably use the application differently. Not only that, but there has to be a consideration for how the user will physically interact with devices such as scrolling, typing, and tapping rather than using a mouse and keyboard. Creating user stories and scenarios will help in this step, and defining requirements will allow you and your team to determine the best way to approach the following steps in this article.
2. Consider usability from a mobile perspective – You want to have a great design for your mobile application, but at the end of the day, customers are going to either stay on your app or leave it based on usability. The UI should be customized for mobile users, which means it’s going to be different than the desktop experience and has to be tested that way. For example, form filling proves to be much more difficult for a mobile user. If a form isn’t optimized correctly, users will often encounter issues where forms are hard to fill out with their touch and keyboard, whereas they don’t have issues on a laptop or desktop computer. Considering how people view and use content on a small screen is key to derive insight for testing. Additionally, interrupt conditions (i.e. what happens when a notification or other actionable pop-up takes you out of your current task) don’t affect desktop users, but they need to be tested on mobile applications to evaluate functionality and performance. These are the components you have to consider that you may not think about when you’re using a laptop computer if you’re testing for mobile users.
Not to mention, testing for UX will become more difficult as mobile devices become more diversified. Device fragmentation is already an obstacle for most testers. In a perfect world, it would be possible to test on every device, browser, operating system, and even screen resolution. In reality, there are thousands of configurations and that just isn’t plausible. Today many phones have similar designs and screen sizes, which means that most people can get an adequate account of their cross-browser compatibility with a few phones. However, with new releases such as the iPhone X and Google Pixel XL, it looks like mobile devices are trending towards bigger screens and narrower bezels, making you wonder what the future of digital media consumption will look like. As designers have to start planning for usability concerns like the iPhone’s notch, testers will be tasked with diversifying their device labs.
3. Utilize your toolkit and testing knowledge – You don’t have to go mobile alone. In fact, there are a plethora of tools specifically to make your life easier when testing. When thinking about open source options, Appium is the way to go for most testers who want to automate. This means that you don’t have to perform the same manual actions over and over again on every new device. But it does take some skills, such as coding. Some testers also use other free tools like Robotium, Selendroid, MonkeyTalk, Calabash, and Frank.
There are also paid options that can help make your testing strategy even better(and faster). TestComplete allows you to create and run repeatable and robust UI tests across native, desktop, and web iOS and Android apps. CrossBrowserTesting, on the other hand, lets you run automated parallel tests across multiple browsers, devices, and OS at once for web testing. But you can’t forget about manual testing either. With over 1,500 browser configurations available for live testing, you can go through and debug on different devices from the comfort of your preferred desktop browser. Additionally, our full-page screenshot comparison tool makes it easy to spot inconsistencies and remotely debug across browsers. And, the best part is that the integration of these tools allows you to use them together.
Of course, deciding when you should test manually and when you should be automating is something you have to use your knowledge of testing best practices. It won’t do you any good to show of your test automation skills without running any exploratory tests first. However, if you depend solely on manual testing you probably won’t get much done in your workday, or if you do it won’t be on very many different devices. Either way, it’s not a great strategy. Look over this resource for a refresher on the best times to test manually vs automating.
4. Pick & prioritize configurations – It’s not just enough to have the right testing tools, you also have to have the right devices to test on. This isn’t a time to trust your intuition — collecting data and analytics on user behavior will be the best way to figure out which screens you should be running your application on.
Google Analytics is a popular method to look at the mobile devices your site’s visitors are using the most. You can use Google Analytics to see what percentage of users are accessing your website on mobile, as well as more specific data such as the device brand, model, service provider, operating system, and screen resolution. You can also use secondary dimensions to see what browsers they’re using. Additionally, you want to make sure you’re testing older browsers that may be more problematic because that’s the point of testing — catching bugs wherever you suspect they may be.
Another way to account for current and potential users is to look at the most popular browsers. You can do this either by region — for example just looking at the United States — or by looking at the most popular configurations worldwide. For example, according to NetMarketShare, Chrome 60, Safari 10, Android Browser 4, Chrome 59, and Chrome 61 are the most popular mobile browsers right now. Testing a number of these popular browsers means you can be sure that you’re targeting the majority of users that will likely land on your website instead of just the people who already have. By looking at both your user data and the most popular configurations, you should be able to get a good handle on where you should be testing.
5. Choose between emulators, simulators, and real devices – Desktop or mobile emulators, simulators, and real devices all have their advantages and disadvantages, but ultimately, they all have their unique purposes and use cases. For the very best mobile test strategy, you should ideally have a mix of emulators, simulators, and real devices. This way, you can approach speed, accuracy, and scale at once.
Browser or mobile device emulators both mimic the functionality of real devices and can imitate a user’s actions. While emulators are a little closer to the real deal since they replicate the software, hardware, and OS, simulators come pretty close to copying the internal behavior, although they do not emulate hardware or work on the OS. While both emulators and simulators are assets for fast mobile testing with unavailable devices, they don’t always encounter the same environmental factors as a real device and therefore aren’t always as accurate because many factors can affect the validity of the results.
Using real devices, on the other hand, means that you’re testing the same way your customers are using their devices, so accuracy is spot-on. This can be especially important when looking at performance or usability. To achieve the precision of real devices, though, you need a real device lab. Whether that be a physical device lab you build yourself or one you access remotely in a cloud is up to you, but it does generally require investing in more costly resources.
6. Make sure your website is truly responsive.
Just because your website loads on a mobile device, doesn’t mean it’s responsive. We often talk about the importance of functional and visual testing, and it’s just as true for mobile. For example, if you have a large header image with text that looks good on a desktop, you might want to do more than just scale it down on mobile. By doing this, you could be compromising the image, which is probably going to be a lot harder to see and read on a small screen. Instead, designing responsively means considering what the site will look like on different screen sizes and, in this case, potentially changing the header image so it’s easier to digest.
This should be accounted for in testing, which means that in addition to diversifying mobile models, browsers, and operating systems, you also want a few different screen sizes to get the best understanding of responsiveness. This might include a tablet, a smaller mobile phone, a larger mobile phone, and even testing devices on both portrait and landscape.
Conclusion
Not just anybody can be a great tester, and similarly, not just anyone can create an effective mobile test strategy. However, if you want the large percentage of users that visit your application on their mobile devices to be able to interact and engage, knowing your stuff when it comes to mobile testing is key.
A mobile testing strategy isn’t built overnight, but understanding the different components will help you start from the ground up. By understanding how to test on mobile devices, which tools to use, and which machines to test on, you can release new integrations with confidence that users won’t be subject to any wacky behavior that desktop users don’t have to deal with.