It’s no secret that testing an application solely on the machine it was developed on isn’t the best way to know whether or not it’s cross-compatible. Once you decide between building or buying a device lab, there still comes the issue of deciding which browsers to test. After all, you can’t test everything.
That's exactly why we're outlining the best methods for picking which browser, operating systems, screen sizes, and devices are going to be most beneficial to your team in testing. By choosing the configurations to account for as many users as possible, you can form an intelligent desktop and mobile testing strategy.
Analytics
Google Analytics is a precise method of finding online user data because it looks at who’s actually coming to your website and how.
To get a list of the devices your users on on with Google Analytics, you’ll want to go to “Audience” on the left side, click down to “Mobile”, and then click “Devices”.
This will show you how many people are visiting your application on what kind of mobile device, and you can get more information by setting a secondary dimension to view additional data such as service provider, operating system, mobile brand, and screen resolutions.
Additionally, you can go to “Overview” under “Mobile” and set the Secondary Dimension to “Browser” to see find out what browsers are being used on desktops, mobile devices, and tablets.
You can also find out what browsers your customers are using by going to “Technology” under “Audience” then clicking “Technology” and “Browser & OS”.
Click “Operating System” as the Primary Dimension and set the Secondary Dimension to “Browser” for the most common configurations. Set the Secondary Dimension to “Browser Version” when on “Browser” as the Primary Dimension for more specific browser data that will tell you how what version users have.
As far as understanding your users goes, this information gives you the most accurate glimpse of the devices you should be testing on. Make a list of some of the most popular browser, operating system, and device combinations to prioritize during testing.
You can also use other analytics engines, such as Mixpanel, to get an understanding of which environments your users are on.
Most Popular Browsers and Devices
While Google Analytics is a great way to understand the current users that come to your website, it’s not very predictive nor does it account for your potential users.
Because of this, it’s a good idea to include a few of the most popular browsers and devices even if they aren’t seen on your analytics list. If a high number of people use them, it’s logical that you want your web application to support them as well.
For example, the top web browsers are Chrome 62, Chrome 61, Safari 11, Safari 10, Firefox 56, IE 11, Firefox 57, and Chrome 56, while the most popular desktop operating systems are Windows 7, Windows 10, Windows XP, Windows 8, Mac 10.12 (Sierra) and Mac 10.11 (El Capitan).
In terms of operating systems for mobile devices, the most popular are Android 6, Android 5, iOS 10, Android 7, Android 4, iOS 10 and iOS 11.
You also may want to think about popularity by regions. If you your organization is international, you will probably want to prioritize Samsung phones when thinking about testing mobile devices, for example, since they lead in worldwide popularity. However, if you’re customer base is mainly located in the United States, you’d want to focus on testing Apple products.
Of course, this will continually change from country to country and might even change month over month, so you will want to continuously check the market share for your target market.
Testing the Old and the New
You also have to keep in mind when looking at analytics and that sometimes they’re more prone to show up in a report because the web application is actually optimized for them, so it makes sense that people will visit your site and come back if the page already works for them.
If you really want to touch all the bases, it's important to include a few old browsers. Older browsers and devices are going to be more problematic, so while it’s good to check popular models, you also want to test on ones that are less popular but still capture a percent of your users.
You also have to consider devices that may be new to the market. While the number of devices you have now may seem like it could get the job done, new devices come out on the time. If your users have them, you want them too.
The new iPhone X is one case where testing would be important. Up until this year, Apple’s iPhones have generally taken on the same design with a few slight changes. The iPhone X was a big deal when announced because of how different it looked due to a larger screen that had a lack of bezels formerly seen on the front.
Additionally, the notch at the top of the screen has proven to be an issue during visual testing, so running your web application on a iPhone X simulator would be a smart move to avoid unfortunate design issues.
Learn from CrossBrowserTesting Customers
Our customers are pros when it comes to cross-browser testing. Recently we looked at some of the most popular browsers, operating systems, and devices our customers have been testing on this year. We found some trends and patterns in this data, as well as some really interesting insights on what they prioritize in testing.
After looking at the millions of browser and operating system configurations our customers have tested in 2018, 100 reign supreme. These results have come from a mix of people in different roles, industries, and company sizes, showing us the most commonly tested configurations out of 1500+ choices.
Check out the full study in What Our Customers' Top 100 Tested Browser and OS Combinations Tell Us.
Whether you’re testing in 5 browsers or 500, keeping in mind some of the most commonly tested configurations will help you outline a strategy for testing.
Problematic Browsers
When thinking about doing a risk analysis of our applications, we want to test pages that get high traffic and business crucial functions, but we also want to test areas that we know are problematic in order to avoid expected downtime. The same is true with browsers.
While you may find that your users are saturated on the most recent versions of Chrome and Safari but you also acknowledge that your application has difficulties in another browser, it's probably a good idea to add that configuration to your mix.
Internet Explorer, for example, may not take up a large portion of your user base, but it’s known to be one of the most problematic browsers. You may understand that Chrome, Safari, or Firefox is a better browsing option, but someone that isn't as familiar with technology might still be using a past version of IE on an older OS, and you may want to check that your web application still works despite their outdated machine.
Even if you aren't running tests on 100 browsers every regression, it's a good idea to do some exploratory testing and visual testing on a wide array of browsers and operating systems in order to get a better understanding of which ones are most likely to create issues for your team.
Sample List
Still don’t know where to start? Choose how many configurations you want to test and follow our testing guide based on these combined factors.
