Would you be surprised if I told you that very few developers test their mobile apps before release? Well, an article by SDTimes back in July stated that only about 8% of developers actually go through this crucial QA process. The article doesn’t state whether or not these developers have testers doing the testing, but based on how many buggy apps I've personally encountered in the mobile marketplace, I wouldn't expect the number to be much higher.
The mindset of testing may be slowly turning a new leaf, but it makes me wonder if these developers would prefer it if their apps were tested. Maybe it’s a resource issue. Maybe it’s that testing on mobile is simply too complicated. Or maybe it’s lack of experience with mobile.
This seems like a pretty reckless practice for developers who take mobile app development seriously. The reason I say this is because mobile apps are put up against a whole new level of scrutiny than apps on the desktop or Web. The "release it now, fix it later" approach doesn't fly on mobile platforms. What’s even worse, a buggy mobile app will cost your company’s credibility and reputation.
Mobile users are unforgiving when it comes to poorly performing applications. The user reviews for mobile marketplace apps prove that users are more likely to remember a company’s defective software than its well-performing app. Customers also won't retry your mobile apps if it caused issues in a previous release. Because of this, apps must at least meet this short list of user requirements:
- be intuitive
- be free from major bugs
- don't be a resource hog
Besides, going through the submission process every time you want to push an update of your mobile app to users is no trivial task.
Nail Down the Moving Parts
It’s obvious that, if you are developing mobile apps for business value, you need to take testing seriously. The issue is that testing on mobile is so much different than other platforms. Testers are at the mercy of moving parts in mobile, and the fragmentation of devices is no help.
All moving parts need to be considered when testing mobile applications. For example:
Fragmentation of Devices–This is the biggest complication in software testing for mobile apps. This has to do with the amount of mobile devices on the market. Every device is different, so testing on as many different devices as possible will have to be a strong consideration. A general rule of thumb is to test on the most popular devices and the devices your target market would be using.
GPS – Does your device require use of the GPS feature? For example, if you’re creating a map app, your app will have to be tested to the responsiveness of the GPS in the device. What would your app do if a GPS signal was lost or couldn’t be obtained?
Data Consumption – Some apps are huge data consumers, and you do not want your app to be the reason someone goes over their data limit. Test to see how much data is being used.
Gestures/Control – What will your customers be doing when using your app. If youare creating an app that tracks your bicycle route, speed, and calories burnt, then you’re going to need to make sure someone can move around the UI with ease. Does your app use the accelerometer of the phone, meaning can you switch the screen between vertical and horizontal?
GUI – Is the app intuitive to use. Does a user need to read a manual in order to get around the app? You will need to make sure the GUI is simple and easy on the eyes or you’ll get users uninstalling it within the first 10 minutes of downloading it.
Processing Power – This has to do with how much processing power and graphics rendering an app needs. This is particularly important for games, but ignore it to your own peril on other apps. The point is that if your app uses too much juice and crashes a user’s device, they won’t be using your app (or any other apps you offer) anytime soon.
Disk Space – Is your app 3MB or 2GB? Most users today won’t have devices with more than 16GB of space, so it may be a good idea to keep your app size at a fraction of that. Most apps I’ve downloaded recently are between 1MB and 10MB. Mobile games will take up more space depending on the size of the game. The largest game I’ve downloaded was around 500MB. Also keep in mind that the users need to use data to download your app, so if your app is big you are going to be eating a big chunk of their monthly data allowance.
Battery– The more resources your app uses on a device, the more battery power you are going to drain. It’s no wonder why gaming on mobile devices drains the batteries so quickly. Your app could also be a resource hog if it performs poorly, so make sure you are running performance tests.
Screen Resolution and Aspect Ratio- The number of different screen resolutions today is 232 and growing. Along with the fragmentation of devices, you now have to deal with the fragmentation of screen sizes on mobile devices. The fact of the matter is, you can't possibly test your mobile apps on every device and every screen, but planning which devices you do test on will make all the difference.
What Type of App Are You Testing?
Next you have to consider what type of mobile app you will be developing.
Native App - An app that only uses the device's hardware.
Web App - An app that uses the mobile browser. There are many mobile Web browsers available, so it will be important to try and test on the ones your target market will be using. Surveying the stakeholders could be a good implication of what your potential users will be using to surf the web on their mobile devices.
Hybrid App - This type of mobile app is a native app that also uses Web technology. This could be data the app pulls on the backend using APIs.
As you can see, strategizing your mobile app testing is no easy feat, and to make matters worse, you’ll find as you approach testing mobile applications that a lot of it is repetitive manual testing or crowd-sourcing. The problem with this is the large fragmentation of mobile hardware.
With new devices being announced almost every day, you’ll find that testing your app on every possible device is not only impractical, but also pretty much impossible. There is just too much variety, whether that may be differences in processing power, Wi-Fi capabilities, memory, aspect ratio, screen resolution, etc. The list goes on, as does the possibility for defects.
Increasing Test Coverage
There are plenty of avenues to increase test coverage of mobile applications, but some are a bit more reliant on outside services than others. Of course, the road you take in testing your mobile apps should ultimately be dependent on the context of what your app does, what type of information will be transferred to and from the device, and how your customers will/need to respond.
For instance, you could use crowd sourcing as a possible means to increase your test coverage, but that requires providing your app to a third party as well as fees that will need to be considered for every test run. You could also test your app using device emulators, which is provided with iOS and Android SDKs (software development kits), but that will never replace the usefulness of real-device testing. Testing must be done to include the variables of reality and not take place in a make-believe fantasy world. You could manually test on as many devices as possible, but this requires a large effort in manpower and time.
What is a Software Tester to do?
An interesting approach which seems to be taking traction slowly is automated mobile testing. Much like automated testing on other platforms, it shouldn’t be your only approach to testing an application since manual testing can’t be replaced, but as we march forward to the future, it’s not so hard to believe that mobile apps are going to get larger in size and more powerful as hardware continues to get smaller.
What can be saved in time and resources with using automated mobile testing is highly beneficial. It also allows you to increase your test coverage since you'll have more time for manually testing your apps. It shouldn't be considered a replacement for manual testing, but rather it complements the manual testing work of the tester, much in the same way it does on other platforms.
By combining these different types of testing (manual and automated) and testing all the moving parts, you increase the chance of your mobile app getting to the marketplace with as few defects as possible. Whatever can be done to increase test coverage quickly is what will be paramount for testing mobile apps. There isn't a right or wrong answer. It’s all about your risk appetite and how much you are willing to spend in terms of time and resources no minimize the likelihood of your app becoming just another “1-star” in the app store.
Do you have any tips on how to help others test mobile apps? Leave a comment below if you do.