Why You Need to Build an Automated Mobile App Testing Matrix
  July 15, 2013

Creating cross-platform mobile development protocols that allow you to play in virtually any OS sandbox is a great idea, but the execution of that is getting increasingly difficult to manage. There are a score of different platforms, OS protocols, form factors and carrier/manufacturer customizations that have to be considered.

Now layer over that mobile Web differences, native apps, or the notorious “all of the above” and you can see that creating an effective, comprehensive and omnibus testing matrix is a marathon. Maybe two.

There are no quick fixes.

If the notion that there is not a quick fix for executing across platforms bothers you, keep in mind that the companies creating proprietary operating systems for their proprietary devices are trying to clobber each other. If they had their way they would run the competition out of town.

Understandably they are not staying awake nights, prostrate with worry over how easy it may be that an app you created for their device can easily be translated to the competition.

Previously, we discussed the merits of device testing vs. emulator testing, coming out strongly in favor of both. In the case of the former we don’t think there’s anything that can replace having the device in hand and actually working with the app. However, having a great app testing framework is also going to require automated abilities as well, for several reasons:

  • It ensures that application features work as designed on the selected mobile devices – not to mention performance on various networks and bandwidths.
  • It tracks stability of the application on desired mobile devices and networks.
  • It reduces how long it takes to get it market ready.
  • The increasing number of mobile applications platforms can be factored in.
  • It gives real-time results on an ongoing basis, simplifying the data retrieval process.

Further, the testing matrix should have the flexibility to go from object testing to element testing. As such, your testing should address three main areas:

First, the number of companies out there making devices, as well as the number of devices your app needs to be available to. This means attention must be paid to rendering differences and run times, with the latter variable between handsets manufactured by the same company.

Second, what are the network types that are going to be brought into play? Within those networks how much customization is present? How do the speeds vary?

Third is to look at the devices and conduct a SWOT (strengths, weaknesses, opportunities, threats) analysis. Some apps may be best used on a device with a large, high-resolution touchscreen. An app that allows the user to reserve a hotel room probably will be wonderful regardless of how much visual real estate a specific mobile phone offers.

And while there is not a one-size-fits-all testing solution coming down the pike that would bring about a huge cost savings when it comes to a superior app testing matrix, consider that once you have one in place it will facilitate your work on future applications going forward.

True, testing your app across all the mobile platforms is never going to be easy as 1-2-3, but setting up, updating and maintaining a mobile app testing matrix isn’t about today. It’s about setting up systems that allow app developers to more easily reach broad audiences with their wares. It’s about building your company’s image in the marketplace as a go-to resource for consumers, no matter which mobile device they may have at any given moment.

It’s a lot of effort, but it’s worth it.

If this sounds like a lot of effort, consider this from The Wall Street Journal:

The odds of striking gold in the apps business are quite long. While there are more than 800,000 mobile apps available in Apple Inc.'s App Store, only 80 of them generated more than $1 million in revenue during the fourth quarter, according to research firm Distimo. Reaching a few hundred downloads a day used to get an app on a Top 10 or 20 list a few years ago. It now requires several thousand, so larger apps companies—such as Electronic Arts Inc., EA Walt Disney Co., Zynga Inc. ZNGA and Rovio Entertainment Ltd.—tend to dominate the top-seller lists. Only 2% of the Top 250 publishers for iPhone apps in the U.S. App Store are newcomers, again according to Distimo.

The question is not whether you should make a substantive investment in a superior testing matrix; the question is why wouldn’t you give your mobile apps every possible chance to succeed?

See also: